Oct 30 12:48:55.032327 ip-10-0-133-61 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 12:48:55.032338 ip-10-0-133-61 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 12:48:55.032347 ip-10-0-133-61 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 12:48:55.032666 ip-10-0-133-61 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 12:49:05.068238 ip-10-0-133-61 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 12:49:05.068253 ip-10-0-133-61 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1acb60c79a59434b89c82519d02f9c6c -- Oct 30 12:50:49.965594 ip-10-0-133-61 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:50:50.337771 ip-10-0-133-61 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:50:50.337771 ip-10-0-133-61 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:50:50.337771 ip-10-0-133-61 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:50:50.337771 ip-10-0-133-61 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:50:50.343091 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343013 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:50:50.343091 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343090 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:50:50.343091 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343093 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343096 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343099 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343102 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343105 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343108 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343111 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343114 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343117 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343120 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343122 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343125 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343128 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343131 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343133 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343137 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343139 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343142 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343145 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343148 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:50:50.343184 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343152 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343154 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343158 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343160 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343163 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343165 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343169 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343172 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343174 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343177 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343180 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343183 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343186 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343188 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343191 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343194 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343197 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343200 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343203 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:50:50.343659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343205 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343208 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343211 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343213 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343216 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343218 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343221 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343224 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343226 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343229 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343231 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343234 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343236 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343239 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343242 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343245 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343248 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343250 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343253 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343256 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:50:50.344112 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343258 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343261 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343264 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343267 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343271 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343273 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343276 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343279 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343282 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343284 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343300 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343303 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343305 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343308 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343311 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343314 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343319 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343323 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343326 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:50:50.344586 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343329 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343332 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343334 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343338 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343340 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343343 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343346 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343349 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343353 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343355 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343358 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343361 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343366 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343370 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343373 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343376 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343380 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343383 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343385 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:50:50.345027 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343388 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343391 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343394 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343398 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343401 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343403 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.343406 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344199 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344207 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344210 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344213 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344216 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344219 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344221 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344224 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344227 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344229 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344232 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344235 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344238 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:50:50.345527 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344242 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344245 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344248 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344251 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344254 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344257 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344260 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344262 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344265 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344268 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344270 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344273 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344276 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344279 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344281 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344284 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344300 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344303 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344306 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:50:50.346010 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344309 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344311 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344314 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344317 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344320 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344323 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344325 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344328 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344331 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344334 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344337 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344339 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344342 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344345 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344348 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344352 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344357 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344361 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344364 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344367 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:50:50.346476 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344370 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344372 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344375 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344377 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344381 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344383 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344386 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344388 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344391 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344394 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344397 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344399 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344402 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344404 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344407 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344410 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344413 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344415 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344418 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344421 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:50:50.346945 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344423 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344426 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344428 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344431 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344433 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344436 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344439 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344441 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344444 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344447 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344449 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344452 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344455 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344457 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344460 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344462 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344465 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344468 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344470 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:50:50.347429 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344473 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344477 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344480 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344483 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344486 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344492 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344495 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344498 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344501 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344504 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344507 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344509 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344513 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344515 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.344518 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345021 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345026 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345029 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345032 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:50:50.347873 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345036 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345040 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345044 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345051 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345054 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345057 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345060 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345063 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345065 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345068 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345071 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345074 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345077 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345080 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345082 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345085 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345088 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345091 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345094 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345097 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:50:50.348324 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345100 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345103 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345105 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345108 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345110 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345113 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345116 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345118 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345121 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345123 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345126 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345129 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345131 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345134 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345137 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345139 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345142 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345145 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345148 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345150 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:50:50.348785 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345153 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345156 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345158 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345161 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345163 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345166 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345168 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345173 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345176 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345178 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345181 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345184 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345186 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345189 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345192 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345194 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345197 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345200 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345202 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:50:50.349348 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345205 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345207 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345210 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345214 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345218 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345221 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345226 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345228 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345231 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345234 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345237 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345240 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345242 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345245 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345247 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345250 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345252 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345255 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345258 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:50:50.349809 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345261 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345265 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345268 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345270 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345273 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345275 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345278 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345281 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345283 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345299 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345302 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345305 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345307 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345310 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345313 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345316 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345318 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345321 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345323 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345326 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:50:50.350280 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345329 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345332 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345335 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.345337 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346415 2569 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346433 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346439 2569 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346444 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346449 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346452 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346458 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346463 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346467 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346473 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346477 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346481 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346484 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346487 2569 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346491 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346494 2569 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346497 2569 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346500 2569 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346505 2569 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346508 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:50:50.350760 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346512 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346515 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346518 2569 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346523 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346526 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346529 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346533 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346536 2569 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346539 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346542 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346545 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346548 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346553 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346556 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346559 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346562 2569 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346565 2569 flags.go:64] FLAG: --enable-server="true" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346568 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346573 2569 flags.go:64] FLAG: --event-burst="100" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346576 2569 flags.go:64] FLAG: --event-qps="50" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346580 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346585 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346588 2569 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346592 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:50:50.351327 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346595 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346598 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346601 2569 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346605 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346608 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346611 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346614 2569 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346617 2569 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346620 2569 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346623 2569 flags.go:64] FLAG: --feature-gates="" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346627 2569 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346631 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346634 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346638 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346641 2569 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346644 2569 flags.go:64] FLAG: --help="false" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346647 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346651 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346654 2569 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346657 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346661 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346664 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346668 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:50:50.351893 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346671 2569 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346674 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346677 2569 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346680 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346683 2569 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346688 2569 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346692 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346695 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346698 2569 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346701 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346704 2569 flags.go:64] FLAG: --lock-file="" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346707 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346710 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346713 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346719 2569 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346722 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346726 2569 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346729 2569 flags.go:64] FLAG: --logging-format="text" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346732 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346736 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346739 2569 flags.go:64] FLAG: --manifest-url="" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346742 2569 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346746 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346750 2569 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346754 2569 flags.go:64] FLAG: --max-pods="110" Oct 30 12:50:50.352470 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346757 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346760 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346763 2569 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346766 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346769 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346772 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346775 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346784 2569 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346787 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346790 2569 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346793 2569 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346796 2569 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346801 2569 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346804 2569 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346807 2569 flags.go:64] FLAG: --port="10250" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346810 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346813 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c12c475088b1ba9a" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346817 2569 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346820 2569 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346823 2569 flags.go:64] FLAG: --register-node="true" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346826 2569 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346830 2569 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346833 2569 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346836 2569 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346838 2569 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346843 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:50:50.353047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346846 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346849 2569 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346852 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346855 2569 flags.go:64] FLAG: --runonce="false" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346858 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346861 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346864 2569 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346867 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346871 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346874 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346877 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346880 2569 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346883 2569 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346887 2569 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346890 2569 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346893 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346897 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346900 2569 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346904 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346909 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346912 2569 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346915 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346920 2569 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346923 2569 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346926 2569 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:50:50.353704 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346929 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346933 2569 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346936 2569 flags.go:64] FLAG: --v="2" Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346944 2569 flags.go:64] FLAG: --version="false" Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346948 2569 flags.go:64] FLAG: --vmodule="" Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346953 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.346957 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347124 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347128 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347131 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347134 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347137 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347140 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347142 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347145 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347148 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347150 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347154 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347158 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347161 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347164 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:50:50.354304 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347166 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347174 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347178 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347180 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347186 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347189 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347191 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347194 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347197 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347199 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347202 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347205 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347208 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347211 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347213 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347216 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347219 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347222 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347224 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:50:50.354916 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347227 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347229 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347232 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347235 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347237 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347240 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347243 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347245 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347248 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347250 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347253 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347255 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347258 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347261 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347263 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347272 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347275 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347280 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347283 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347286 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:50:50.355659 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347302 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347305 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347313 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347316 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347319 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347321 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347324 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347327 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347329 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347332 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347335 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347338 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347340 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347343 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347345 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347348 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347350 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347353 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347356 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:50:50.356259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347358 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347361 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347364 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347367 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347369 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347373 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347375 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347378 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347381 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347390 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347395 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347398 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347400 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347403 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347406 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347408 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347411 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347414 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347416 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347419 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:50:50.356791 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347422 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347424 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347427 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347431 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347435 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347438 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347441 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347444 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347447 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347449 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347452 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347454 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347457 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347459 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.347486 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:50:50.357259 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347532 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347538 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347541 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347544 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347547 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347549 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347552 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347561 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347566 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347569 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347571 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347574 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347577 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347580 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347582 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347585 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347588 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347591 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347593 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347596 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:50:50.357642 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347599 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347602 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347606 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347609 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347612 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347615 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347617 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347620 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347623 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347625 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347628 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347631 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347633 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347636 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347639 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347642 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347644 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347647 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347650 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:50:50.358115 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347654 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347665 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347669 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347671 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347674 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347677 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347679 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347682 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347685 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347687 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347690 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347693 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347695 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347698 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347701 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347703 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347706 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347708 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347711 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347714 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:50:50.358577 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347723 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347726 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347728 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347731 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347734 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347737 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347740 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347742 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347745 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347747 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347750 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347753 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347757 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347761 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347769 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347772 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347775 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347777 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347780 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:50:50.359045 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347782 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347785 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347788 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347790 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347793 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347796 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347798 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347801 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347804 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347806 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347809 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347812 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347814 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347817 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347820 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347822 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347825 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347828 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347830 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:50:50.359567 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347833 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347835 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347838 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347841 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347843 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347846 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347854 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347858 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.347861 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.357448 2569 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.357465 2569 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.357482 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.357488 2569 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357559 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357565 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357570 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357575 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357579 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357582 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:50:50.360011 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357585 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357589 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357592 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357594 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357598 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357602 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357606 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357609 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357612 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357615 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357617 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357620 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357623 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357625 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357628 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357631 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357634 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357636 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357639 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:50:50.360541 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357642 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357644 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357648 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357652 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357655 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357658 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357661 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357664 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357667 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357669 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357672 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357675 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357678 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357680 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357683 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357686 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357689 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357691 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357694 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357697 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:50:50.360976 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357700 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357703 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357706 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357709 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357712 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357714 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357717 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357720 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357722 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357725 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357728 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357730 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357733 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357736 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357739 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357742 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357745 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357747 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357750 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357753 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:50:50.361484 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357756 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357758 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357761 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357763 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357766 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357769 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357772 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357774 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357777 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357780 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357782 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357785 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357787 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357790 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357794 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357796 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357799 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357802 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357804 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:50:50.361949 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357807 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357810 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357812 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357815 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357818 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357821 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357824 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357827 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357830 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357832 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357835 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357838 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357840 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357843 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357846 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357848 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357851 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357854 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357857 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357859 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:50:50.362423 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357862 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.357864 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.357895 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358042 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358047 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358050 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358053 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358056 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358059 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358062 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358064 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358067 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358070 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358073 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358075 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358078 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:50:50.362874 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358081 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358083 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358086 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358088 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358091 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358094 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358096 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358099 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358101 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358104 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358107 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358109 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358112 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358115 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358118 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358123 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358126 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358129 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358132 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:50:50.363252 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358135 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358138 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358141 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358144 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358146 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358149 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358152 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358155 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358158 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358160 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358163 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358166 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358168 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358171 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358173 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358176 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358179 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358181 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358184 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:50:50.363711 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358187 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358189 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358192 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358195 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358197 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358201 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358205 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358209 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358212 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358215 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358218 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358220 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358223 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358226 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358229 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358231 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358234 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358237 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358239 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:50:50.364147 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358242 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358245 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358248 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358250 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358253 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358255 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358258 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358260 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358263 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358265 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358268 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358271 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358274 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358276 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358279 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358281 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358284 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358304 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358309 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:50:50.364605 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358312 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358315 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358318 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358321 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358324 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358326 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358329 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358332 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358335 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358338 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358340 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358343 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358346 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358348 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358351 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358354 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:50:50.365035 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358357 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.358382 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358415 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358419 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358422 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358425 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358427 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358430 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358433 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358436 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358439 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358441 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358444 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358447 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358449 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358452 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:50:50.365434 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358456 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358459 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358462 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358465 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358467 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358473 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358476 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358479 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358481 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358484 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358487 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358490 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358492 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358495 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358498 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358501 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358504 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358507 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358510 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:50:50.365844 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358512 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358515 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358518 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358520 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358523 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358526 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358528 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358531 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358534 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358536 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358539 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358542 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358545 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358548 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358551 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358554 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358557 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358559 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358563 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358566 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:50:50.366331 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358569 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358572 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358574 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358577 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358580 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358582 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358585 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358589 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358592 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358594 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358598 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358601 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358603 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358606 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358609 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358611 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358614 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358617 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358620 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358624 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:50:50.366827 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358628 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358631 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358634 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358637 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358639 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358644 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358646 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358649 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358652 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358655 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358661 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358664 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358666 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358669 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358672 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358675 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358677 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358680 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358683 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358685 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:50:50.367367 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358688 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358690 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358693 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358697 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358700 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358702 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358705 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358708 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358712 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358716 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358719 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358724 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:50.358727 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.358787 2569 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.362237 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.363343 2569 server.go:1014] "Starting client certificate rotation" Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.363436 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.364301 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:50:50.367861 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.365580 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:50:50.368451 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.366795 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:50:50.375040 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.375026 2569 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:50:50.378838 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.378823 2569 log.go:25] "Validated CRI v1 image API" Oct 30 12:50:50.379702 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.379674 2569 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:50:50.385528 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.385503 2569 fs.go:135] Filesystem UUIDs: map[5da986bb-eb87-4471-8b83-e126387a21ba:/dev/nvme0n1p4 65cbc65c-dc77-4a3f-bc0f-576f44b51184:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 12:50:50.385615 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.385527 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 12:50:50.392585 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.392458 2569 manager.go:217] Machine: {Timestamp:2026-10-30 12:50:50.391441736 +0000 UTC m=+0.332161955 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3071742 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23efe48c8cced7db791e27285b59cc SystemUUID:ec23efe4-8c8c-ced7-db79-1e27285b59cc BootID:1acb60c7-9a59-434b-89c8-2519d02f9c6c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:1b:20:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:1b:20:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:9c:6b:4a:9e:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:50:50.392585 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.392569 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:50:50.392711 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.392683 2569 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:50:50.394001 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.393975 2569 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:50:50.394137 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.394004 2569 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-61.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:50:50.394178 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.394139 2569 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:50:50.394178 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.394146 2569 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:50:50.394178 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.394170 2569 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:50:50.395913 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.395902 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:50:50.397197 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.397187 2569 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:50:50.402682 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.402668 2569 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:50:50.402801 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.402792 2569 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:50:50.403616 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.403608 2569 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:50:50.403649 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.403620 2569 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:50:50.403649 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.403632 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:50:50.403649 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.403642 2569 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:50:50.403751 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.403651 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:50:50.403988 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.403976 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:50:50.406596 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.406580 2569 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:50:50.413949 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.413934 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:50:50.414884 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.414872 2569 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:50:50.414934 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.414902 2569 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:50:50.415099 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415084 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:50:50.415127 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415110 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:50:50.415127 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415120 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:50:50.415127 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415126 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:50:50.415203 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415133 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:50:50.415203 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415138 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:50:50.415773 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415764 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:50:50.415808 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415774 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:50:50.415808 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415781 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:50:50.415808 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415787 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:50:50.415808 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415796 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:50:50.415808 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.415806 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:50:50.417207 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.417197 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:50:50.417242 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.417210 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:50:50.421352 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.421330 2569 server.go:1263] "Started kubelet" Oct 30 12:50:50.421533 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.421402 2569 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:50:50.421594 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.421484 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:50:50.421677 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.421665 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:50:50.422250 ip-10-0-133-61 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:50:50.425464 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.425185 2569 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:50:50.427337 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.427317 2569 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:50:50.429312 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.429282 2569 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 12:50:50.433581 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.433541 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:50:50.434871 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.434854 2569 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:50:50.435073 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.435054 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:50.435738 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.435721 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:50:50.435738 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.435722 2569 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:50:50.435858 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.435747 2569 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:50:50.435858 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.435810 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:50:50.435858 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.435825 2569 factory.go:55] Registering systemd factory Oct 30 12:50:50.435858 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.435837 2569 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:50:50.435858 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.435853 2569 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:50:50.435858 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.435858 2569 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:50:50.436106 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.436091 2569 factory.go:153] Registering CRI-O factory Oct 30 12:50:50.436160 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.436112 2569 factory.go:223] Registration of the crio container factory successfully Oct 30 12:50:50.436160 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.436134 2569 factory.go:103] Registering Raw factory Oct 30 12:50:50.436160 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.436149 2569 manager.go:1199] Started watching for new ooms in manager Oct 30 12:50:50.437251 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.437238 2569 manager.go:319] Starting recovery of all containers Oct 30 12:50:50.448891 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.448875 2569 manager.go:324] Recovery completed Oct 30 12:50:50.454674 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.454660 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:50:50.455244 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.455085 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:50:50.456392 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.456362 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:50:50.456476 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.456401 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:50:50.456476 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.456413 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:50:50.457065 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.457049 2569 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:50:50.457127 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.457080 2569 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:50:50.457127 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.457101 2569 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:50:50.457127 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.457113 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:50:50.457806 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.457791 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:50:50.457855 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.457817 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:50:50.457855 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.457827 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:50:50.459384 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.459372 2569 policy_none.go:50] "Start" Oct 30 12:50:50.459420 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.459391 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:50:50.459420 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.459401 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:50:50.460388 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.460333 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-61.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 12:50:50.461452 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.461436 2569 policy_none.go:44] "Start" Oct 30 12:50:50.461534 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.460341 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-61.ec2.internal.18e34fa5dd1a717f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-61.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-61.ec2.internal,},FirstTimestamp:2026-10-30 12:50:50.421285247 +0000 UTC m=+0.362005465,LastTimestamp:2026-10-30 12:50:50.421285247 +0000 UTC m=+0.362005465,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-61.ec2.internal,}" Oct 30 12:50:50.473101 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.473080 2569 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 12:50:50.480560 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.480537 2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 12:50:50.480752 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.480687 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-61.ec2.internal.18e34fa5df32161b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-61.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-61.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-61.ec2.internal,},FirstTimestamp:2026-10-30 12:50:50.456389147 +0000 UTC m=+0.397109365,LastTimestamp:2026-10-30 12:50:50.456389147 +0000 UTC m=+0.397109365,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-61.ec2.internal,}" Oct 30 12:50:50.484739 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.484719 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xw2qt" Oct 30 12:50:50.490261 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.490231 2569 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 12:50:50.490471 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.490284 2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 12:50:50.513164 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.496688 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xw2qt" Oct 30 12:50:50.513164 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.507487 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:50:50.513164 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.508768 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:50:50.513164 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.508786 2569 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:50:50.513164 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.508801 2569 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:50:50.513164 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.508834 2569 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 12:50:50.513164 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.510877 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-61.ec2.internal.18e34fa5df325aa0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-61.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-61.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-61.ec2.internal,},FirstTimestamp:2026-10-30 12:50:50.456406688 +0000 UTC m=+0.397126905,LastTimestamp:2026-10-30 12:50:50.456406688 +0000 UTC m=+0.397126905,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-61.ec2.internal,}" Oct 30 12:50:50.535360 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.535334 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:50.536413 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.536399 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:50:50.609067 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.609045 2569 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 12:50:50.636008 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.635989 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:50.639847 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.639823 2569 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:50:50.642075 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.642060 2569 manager.go:341] "Starting Device Plugin manager" Oct 30 12:50:50.642160 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.642118 2569 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:50:50.642160 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.642134 2569 server.go:87] "Starting device plugin registration server" Oct 30 12:50:50.642534 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.642520 2569 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:50:50.642615 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.642534 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:50:50.642666 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.642634 2569 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:50:50.642744 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.642729 2569 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:50:50.642817 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.642745 2569 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:50:50.643396 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.643363 2569 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:50:50.643464 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.643414 2569 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:50.703790 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.703766 2569 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-61.ec2.internal\" not found" node="ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.743209 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.743184 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:50:50.744452 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.744437 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:50:50.744534 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.744467 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:50:50.744534 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.744476 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:50:50.744534 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.744500 2569 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.766718 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.766694 2569 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.766850 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.766724 2569 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-61.ec2.internal\": node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:50.809218 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.809175 2569 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal"] Oct 30 12:50:50.809373 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.809253 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:50:50.810282 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.810265 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:50:50.810392 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.810310 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:50:50.810392 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.810322 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:50:50.812644 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.812632 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:50:50.812811 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.812797 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.812846 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.812826 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:50:50.813457 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.813434 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:50:50.813528 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.813459 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:50:50.813528 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.813473 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:50:50.813528 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.813435 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:50:50.813629 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.813542 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:50:50.813629 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.813560 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:50:50.815657 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.815640 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.815740 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.815664 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:50:50.816422 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.816398 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:50:50.816551 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.816428 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:50:50.816551 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.816442 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:50:50.839083 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.839062 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.839173 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.839090 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.839173 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.839114 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bac4afb01d6e1d9bf4827fcda4b8b291-config\") pod \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" (UID: \"bac4afb01d6e1d9bf4827fcda4b8b291\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.845833 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.845817 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-61.ec2.internal\" not found" node="ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.848434 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.848416 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-61.ec2.internal\" not found" node="ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.864383 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.864335 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:50.939966 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.939939 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.939966 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.939969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.940139 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.939987 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bac4afb01d6e1d9bf4827fcda4b8b291-config\") pod \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" (UID: \"bac4afb01d6e1d9bf4827fcda4b8b291\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.940139 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.940044 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bac4afb01d6e1d9bf4827fcda4b8b291-config\") pod \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" (UID: \"bac4afb01d6e1d9bf4827fcda4b8b291\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.940139 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.940054 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.940139 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:50.940054 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:50:50.965073 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:50.965045 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:51.065782 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:51.065757 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:52.513386 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:52.513312 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:50:52.515451 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:52.515433 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:50:52.528091 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:52.528076 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:52.628820 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:52.628772 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:52.725249 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:52.725219 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 12:50:52.725753 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:52.725385 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 12:50:52.729455 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:52.729431 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:52.830010 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:52.829982 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:52.852397 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:52.852380 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:50:52.854781 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:52.854764 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:50:52.854868 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:52.854796 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:50:52.854868 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:52.854808 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:50:52.864844 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:52.864807 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:45:51 +0000 UTC" deadline="2028-04-11 18:52:12.869154324 +0000 UTC" Oct 30 12:50:52.864844 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:52.864841 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12702h1m20.004316726s" Oct 30 12:50:52.930397 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:52.930364 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:53.031015 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:53.030984 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:53.102271 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:53.102186 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a61b1589e9c4b5de6634f4c13443710.slice/crio-7d67e7e634aaf4db20f4c3d88b3b869f823864006342c66cd3918339f37396b7 WatchSource:0}: Error finding container 7d67e7e634aaf4db20f4c3d88b3b869f823864006342c66cd3918339f37396b7: Status 404 returned error can't find the container with id 7d67e7e634aaf4db20f4c3d88b3b869f823864006342c66cd3918339f37396b7 Oct 30 12:50:53.107862 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:53.107840 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac4afb01d6e1d9bf4827fcda4b8b291.slice/crio-f40308f148529bb5b59cc5f21ec25642c8f89bc01f8f39ecddf49e7f2dfcc3c6 WatchSource:0}: Error finding container f40308f148529bb5b59cc5f21ec25642c8f89bc01f8f39ecddf49e7f2dfcc3c6: Status 404 returned error can't find the container with id f40308f148529bb5b59cc5f21ec25642c8f89bc01f8f39ecddf49e7f2dfcc3c6 Oct 30 12:50:53.108138 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.108120 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:50:53.128932 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.128914 2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:50:53.132020 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:53.132006 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:53.232206 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:53.232180 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:50:53.245519 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.245502 2569 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:50:53.297701 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.297676 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:50:53.326926 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.326908 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:50:53.328999 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.328985 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:50:53.346884 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.346817 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:50:53.456621 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.456595 2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:50:53.767101 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.766653 2569 apiserver.go:52] "Watching apiserver" Oct 30 12:50:53.793607 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.793582 2569 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:50:53.794007 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.793982 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-cdptr","kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2ktf5","openshift-network-diagnostics/network-check-target-b9v2t","openshift-image-registry/node-ca-6p76v","openshift-multus/multus-s78dq","openshift-ovn-kubernetes/ovnkube-node-gp5jc","kube-system/global-pull-secret-syncer-jtc6z","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx","openshift-dns/node-resolver-v4l8n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal","openshift-multus/multus-additional-cni-plugins-dnwpp","openshift-multus/network-metrics-daemon-l7pxj","openshift-network-operator/iptables-alerter-n4rct"] Oct 30 12:50:53.796462 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.796444 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 12:50:53.797021 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.797004 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.799097 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.799075 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:53.799214 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:53.799195 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:50:53.801416 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.801395 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.803774 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.803673 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:50:53.803774 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:53.803736 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:50:53.806052 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.806028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-ovn\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809614 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-node-log\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809685 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809690 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-script-lib\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809729 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-netns\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809756 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-systemd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-netd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809817 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2767673-ded2-4026-8b3f-ff2582c0aede-ovn-node-metrics-cert\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-slash\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809903 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-var-lib-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.809990 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-log-socket\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810048 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810065 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-config\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810390 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx6xd\" (UniqueName: \"kubernetes.io/projected/b2767673-ded2-4026-8b3f-ff2582c0aede-kube-api-access-nx6xd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.811182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlchw\" (UniqueName: \"kubernetes.io/projected/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-kube-api-access-qlchw\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:53.812093 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-bin\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.812093 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-env-overrides\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.812093 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810553 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:53.812093 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810584 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-kubelet\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.812093 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810627 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-systemd-units\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.812093 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.810656 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-etc-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.812093 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.811522 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:53.813042 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.812823 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:50:53.813042 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.812898 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:50:53.813042 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.812966 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d8ng4\"" Oct 30 12:50:53.816279 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.816246 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:50:53.817453 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.817434 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:53.817992 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.817975 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:50:53.818749 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.818732 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:53.821542 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.821526 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:53.821640 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:53.821592 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:50:53.823929 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.823911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:53.853911 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.853886 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:50:53.855551 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.853992 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zxpw6" Oct 30 12:50:53.858067 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.858047 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zxpw6" Oct 30 12:50:53.875655 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.875557 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" event={"ID":"bac4afb01d6e1d9bf4827fcda4b8b291","Type":"ContainerStarted","Data":"f40308f148529bb5b59cc5f21ec25642c8f89bc01f8f39ecddf49e7f2dfcc3c6"} Oct 30 12:50:53.878071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.878031 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" event={"ID":"9a61b1589e9c4b5de6634f4c13443710","Type":"ContainerStarted","Data":"7d67e7e634aaf4db20f4c3d88b3b869f823864006342c66cd3918339f37396b7"} Oct 30 12:50:53.899071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.899052 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:50:53.911503 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911464 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:53.911608 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911512 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-node-log\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.911649 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911620 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.911649 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911629 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-node-log\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.911723 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-tuned\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.911761 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:53.911799 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911764 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:53.911830 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911795 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.911898 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911878 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-systemd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.912006 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911935 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-systemd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.912006 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.911967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxdg\" (UniqueName: \"kubernetes.io/projected/28a37fed-2af0-45d1-8e84-c658dbff6b61-kube-api-access-zlxdg\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:53.912106 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912032 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-lib-modules\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.912106 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-config\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.912210 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx6xd\" (UniqueName: \"kubernetes.io/projected/b2767673-ded2-4026-8b3f-ff2582c0aede-kube-api-access-nx6xd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.912210 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912129 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-sys-fs\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:53.912567 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912539 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-agent-certs\") pod \"konnectivity-agent-cdptr\" (UID: \"b0aa96f3-47c7-48f1-98a3-dd3c4548d593\") " pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:50:53.912666 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912579 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28a37fed-2af0-45d1-8e84-c658dbff6b61-iptables-alerter-script\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:53.912666 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912625 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0471989-f151-47a5-9d39-97d85760da9e-tmp-dir\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:53.912666 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-cnibin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.912666 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912658 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-k8s-cni-cncf-io\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.912889 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-bin\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.912889 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912734 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-multus\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.912889 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912749 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-daemon-config\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.912889 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912765 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-systemd-units\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.912889 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912780 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-etc-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.912889 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:53.912889 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-run\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.912889 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912849 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-netns\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912945 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-device-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-conf\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912982 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-var-lib-kubelet\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.912997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-netns\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913010 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-hostroot\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913046 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-var-lib-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913076 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlchw\" (UniqueName: \"kubernetes.io/projected/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-kube-api-access-qlchw\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913089 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-systemd\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913109 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvtd\" (UniqueName: \"kubernetes.io/projected/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-kube-api-access-ccvtd\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:53.913145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913140 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-registration-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysconfig\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913190 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913219 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-socket-dir-parent\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913361 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-multus-certs\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913379 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cnibin\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913396 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-ovn\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913410 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-script-lib\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913453 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0471989-f151-47a5-9d39-97d85760da9e-hosts-file\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913482 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7t4r\" (UniqueName: \"kubernetes.io/projected/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-kube-api-access-h7t4r\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913500 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-bin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913527 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2767673-ded2-4026-8b3f-ff2582c0aede-ovn-node-metrics-cert\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-slash\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.913589 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913594 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-host\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913611 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-system-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913635 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913668 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-env-overrides\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-modprobe-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-cni-binary-copy\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913795 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-etc-kubernetes\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913829 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-dbus\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcd62\" (UniqueName: \"kubernetes.io/projected/126669c1-ee47-49ab-a6f3-3592a66b1fd2-kube-api-access-dcd62\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-kubelet\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913878 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-socket-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913907 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-konnectivity-ca\") pod \"konnectivity-agent-cdptr\" (UID: \"b0aa96f3-47c7-48f1-98a3-dd3c4548d593\") " pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913925 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.913948 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-conf-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.914147 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.914130 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:50:53.914794 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.914750 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g6csn\"" Oct 30 12:50:53.915070 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.915045 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:50:53.915162 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.915147 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:50:53.915251 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.915241 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:50:53.915516 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.915495 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:50:53.915751 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.915738 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:50:53.916022 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.916008 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fv4k6\"" Oct 30 12:50:53.916204 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.916193 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:50:53.916280 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.916270 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:50:53.916550 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.916536 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:50:53.916998 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.916981 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:50:53.917090 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.917078 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.917404 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.918373 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.918576 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.918812 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/126669c1-ee47-49ab-a6f3-3592a66b1fd2-serviceca\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.918849 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-netd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.918886 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a37fed-2af0-45d1-8e84-c658dbff6b61-host-slash\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.918920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-kubernetes\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.918952 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-tmp\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.918998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r774z\" (UniqueName: \"kubernetes.io/projected/15a59c67-94e0-48e5-b8ff-28a55dea0996-kube-api-access-r774z\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-kubelet-config\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-log-socket\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919136 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s86zc\" (UniqueName: \"kubernetes.io/projected/d0471989-f151-47a5-9d39-97d85760da9e-kube-api-access-s86zc\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-sys\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919198 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-kubelet\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919232 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-system-cni-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919261 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/126669c1-ee47-49ab-a6f3-3592a66b1fd2-host\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:53.920782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919300 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919348 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f27s\" (UniqueName: \"kubernetes.io/projected/f0297b8f-b8e8-470e-a0cb-838697059ff4-kube-api-access-5f27s\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919439 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-netns\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.919482 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-bin\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.920084 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-config\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.920174 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-systemd-units\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.920216 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-etc-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.920275 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-var-lib-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.920610 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-kubelet\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:53.920910 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.921027 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-env-overrides\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:53.921072 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs podName:b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:54.421041002 +0000 UTC m=+2.999730565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs") pod "network-metrics-daemon-l7pxj" (UID: "b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.921125 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-ovn\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.921234 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.921301 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-log-socket\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.921675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.921395 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-slash\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.922689 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.922068 2569 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:50:53.922689 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.922394 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-netd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.923018 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.920314 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.925668 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.925715 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.925747 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.925793 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hfgh7\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.925844 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.925905 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.925953 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9fz8p\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.925982 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.926029 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-78z4l\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.926068 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-588rd\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.926124 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nklv7\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.926180 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.926228 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pdl9w\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.926260 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.926380 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:50:53.926449 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.926411 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:50:53.927122 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.927104 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2767673-ded2-4026-8b3f-ff2582c0aede-ovn-node-metrics-cert\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.932399 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.932378 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-script-lib\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.935969 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.935946 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx6xd\" (UniqueName: \"kubernetes.io/projected/b2767673-ded2-4026-8b3f-ff2582c0aede-kube-api-access-nx6xd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:53.943463 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:53.943436 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlchw\" (UniqueName: \"kubernetes.io/projected/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-kube-api-access-qlchw\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:54.020501 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020433 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.020501 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020475 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r774z\" (UniqueName: \"kubernetes.io/projected/15a59c67-94e0-48e5-b8ff-28a55dea0996-kube-api-access-r774z\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.020501 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-kubelet-config\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:54.020752 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86zc\" (UniqueName: \"kubernetes.io/projected/d0471989-f151-47a5-9d39-97d85760da9e-kube-api-access-s86zc\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:54.020752 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020556 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-sys\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.020752 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-kubelet\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.020752 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020610 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-system-cni-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.020752 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020633 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/126669c1-ee47-49ab-a6f3-3592a66b1fd2-host\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:54.020752 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020671 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-sys\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.020752 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f27s\" (UniqueName: \"kubernetes.io/projected/f0297b8f-b8e8-470e-a0cb-838697059ff4-kube-api-access-5f27s\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.020752 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.020752 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020745 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-tuned\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020767 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020822 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxdg\" (UniqueName: \"kubernetes.io/projected/28a37fed-2af0-45d1-8e84-c658dbff6b61-kube-api-access-zlxdg\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020849 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-lib-modules\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020876 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-sys-fs\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020921 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-kubelet\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020932 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-sys-fs\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020632 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.020970 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-system-cni-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021022 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-kubelet-config\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:54.021153 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021149 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-lib-modules\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021169 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021211 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-agent-certs\") pod \"konnectivity-agent-cdptr\" (UID: \"b0aa96f3-47c7-48f1-98a3-dd3c4548d593\") " pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021239 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28a37fed-2af0-45d1-8e84-c658dbff6b61-iptables-alerter-script\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0471989-f151-47a5-9d39-97d85760da9e-tmp-dir\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021275 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-cnibin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021289 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-k8s-cni-cncf-io\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-multus\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021358 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-daemon-config\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021396 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021398 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-k8s-cni-cncf-io\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021418 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-run\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021444 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-device-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021494 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-cnibin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021504 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-device-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-conf\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021552 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-multus\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021598 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0471989-f151-47a5-9d39-97d85760da9e-tmp-dir\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:54.021671 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-var-lib-kubelet\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021667 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-conf\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021706 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/126669c1-ee47-49ab-a6f3-3592a66b1fd2-host\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021757 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-run\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-var-lib-kubelet\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.021592 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-netns\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022088 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022155 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-hostroot\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022169 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-netns\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022230 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-hostroot\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-systemd\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022279 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvtd\" (UniqueName: \"kubernetes.io/projected/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-kube-api-access-ccvtd\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022306 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-registration-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022340 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-systemd\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022363 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysconfig\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022388 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022411 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-registration-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.022465 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-socket-dir-parent\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022452 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-multus-certs\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022467 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-socket-dir-parent\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022497 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cnibin\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0471989-f151-47a5-9d39-97d85760da9e-hosts-file\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022553 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-multus-certs\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022602 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cnibin\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022636 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022648 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0471989-f151-47a5-9d39-97d85760da9e-hosts-file\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7t4r\" (UniqueName: \"kubernetes.io/projected/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-kube-api-access-h7t4r\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022534 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysconfig\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-bin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022743 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-bin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022761 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-host\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-system-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022818 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022862 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.023286 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.022900 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022925 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-host\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.022952 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28a37fed-2af0-45d1-8e84-c658dbff6b61-iptables-alerter-script\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023023 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-system-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.023038 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret podName:00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:54.523022247 +0000 UTC m=+3.101711798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret") pod "global-pull-secret-syncer-jtc6z" (UID: "00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023053 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-modprobe-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023087 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-cni-binary-copy\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-etc-kubernetes\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023121 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-modprobe-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023133 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-dbus\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023194 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd62\" (UniqueName: \"kubernetes.io/projected/126669c1-ee47-49ab-a6f3-3592a66b1fd2-kube-api-access-dcd62\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023230 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-socket-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023234 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-etc-kubernetes\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-konnectivity-ca\") pod \"konnectivity-agent-cdptr\" (UID: \"b0aa96f3-47c7-48f1-98a3-dd3c4548d593\") " pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023285 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:50:54.024119 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023310 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-conf-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023358 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023350 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/126669c1-ee47-49ab-a6f3-3592a66b1fd2-serviceca\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023442 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-conf-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023461 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a37fed-2af0-45d1-8e84-c658dbff6b61-host-slash\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023497 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-kubernetes\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023519 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-tmp\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023524 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-dbus\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023608 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-kubernetes\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023613 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-socket-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023647 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a37fed-2af0-45d1-8e84-c658dbff6b61-host-slash\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023836 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/126669c1-ee47-49ab-a6f3-3592a66b1fd2-serviceca\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:54.024721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.023887 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-konnectivity-ca\") pod \"konnectivity-agent-cdptr\" (UID: \"b0aa96f3-47c7-48f1-98a3-dd3c4548d593\") " pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:50:54.025193 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.024782 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-tuned\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.025193 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.024978 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-daemon-config\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.025193 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.024989 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-cni-binary-copy\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.026050 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.026031 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-tmp\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.026839 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.026818 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-agent-certs\") pod \"konnectivity-agent-cdptr\" (UID: \"b0aa96f3-47c7-48f1-98a3-dd3c4548d593\") " pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:50:54.045982 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.045960 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:50:54.045982 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.045985 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:50:54.046141 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.045994 2569 projected.go:196] Error preparing data for projected volume kube-api-access-pm47z for pod openshift-network-diagnostics/network-check-target-b9v2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:50:54.046141 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.046129 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z podName:b6151a5c-6392-4151-8c74-6d518f34b4d0 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:54.546105026 +0000 UTC m=+3.124794593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pm47z" (UniqueName: "kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z") pod "network-check-target-b9v2t" (UID: "b6151a5c-6392-4151-8c74-6d518f34b4d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:50:54.047544 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.047493 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r774z\" (UniqueName: \"kubernetes.io/projected/15a59c67-94e0-48e5-b8ff-28a55dea0996-kube-api-access-r774z\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.047979 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.047954 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86zc\" (UniqueName: \"kubernetes.io/projected/d0471989-f151-47a5-9d39-97d85760da9e-kube-api-access-s86zc\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:54.050446 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.050405 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7t4r\" (UniqueName: \"kubernetes.io/projected/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-kube-api-access-h7t4r\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.052290 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.052267 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvtd\" (UniqueName: \"kubernetes.io/projected/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-kube-api-access-ccvtd\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.052290 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.052280 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f27s\" (UniqueName: \"kubernetes.io/projected/f0297b8f-b8e8-470e-a0cb-838697059ff4-kube-api-access-5f27s\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.053528 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.053506 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxdg\" (UniqueName: \"kubernetes.io/projected/28a37fed-2af0-45d1-8e84-c658dbff6b61-kube-api-access-zlxdg\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:54.053626 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.053607 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcd62\" (UniqueName: \"kubernetes.io/projected/126669c1-ee47-49ab-a6f3-3592a66b1fd2-kube-api-access-dcd62\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:54.120610 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.120579 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:50:54.129796 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.129773 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:50:54.133668 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.133644 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:50:54.137554 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.137392 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:50:54.141819 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:54.141784 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b70bdc_18a4_4b6f_89c3_2ed5dc1e8bb3.slice/crio-bbc64b3d91231d34f5dc0e5857e20f292310dcd87c48b60ca0303964ef8f18de WatchSource:0}: Error finding container bbc64b3d91231d34f5dc0e5857e20f292310dcd87c48b60ca0303964ef8f18de: Status 404 returned error can't find the container with id bbc64b3d91231d34f5dc0e5857e20f292310dcd87c48b60ca0303964ef8f18de Oct 30 12:50:54.155376 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.155351 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s78dq" Oct 30 12:50:54.157424 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.157404 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:50:54.165261 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.165239 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:50:54.165938 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:54.165912 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a59c67_94e0_48e5_b8ff_28a55dea0996.slice/crio-60edcebe872bf55f027cc442100417d57793ec9b989f58f1b541cdb7ecec3b2e WatchSource:0}: Error finding container 60edcebe872bf55f027cc442100417d57793ec9b989f58f1b541cdb7ecec3b2e: Status 404 returned error can't find the container with id 60edcebe872bf55f027cc442100417d57793ec9b989f58f1b541cdb7ecec3b2e Oct 30 12:50:54.167083 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.167040 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:50:54.175789 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.175770 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:50:54.178135 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:54.178112 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0297b8f_b8e8_470e_a0cb_838697059ff4.slice/crio-3956a38b2219f01a26c99d1ea7cb75dfeaefd9601fc67f4398a1465d239d0062 WatchSource:0}: Error finding container 3956a38b2219f01a26c99d1ea7cb75dfeaefd9601fc67f4398a1465d239d0062: Status 404 returned error can't find the container with id 3956a38b2219f01a26c99d1ea7cb75dfeaefd9601fc67f4398a1465d239d0062 Oct 30 12:50:54.178821 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:54.178796 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0aa96f3_47c7_48f1_98a3_dd3c4548d593.slice/crio-027a98a5d9752e6e47d1a3545787666832bc6f7d9dc2a465428d4c3e538577f9 WatchSource:0}: Error finding container 027a98a5d9752e6e47d1a3545787666832bc6f7d9dc2a465428d4c3e538577f9: Status 404 returned error can't find the container with id 027a98a5d9752e6e47d1a3545787666832bc6f7d9dc2a465428d4c3e538577f9 Oct 30 12:50:54.179503 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.179363 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:50:54.188652 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:54.188589 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a37fed_2af0_45d1_8e84_c658dbff6b61.slice/crio-cc486ad4df2adafbc04482c3cc486169273ab8751d5532c121849215ff6bef61 WatchSource:0}: Error finding container cc486ad4df2adafbc04482c3cc486169273ab8751d5532c121849215ff6bef61: Status 404 returned error can't find the container with id cc486ad4df2adafbc04482c3cc486169273ab8751d5532c121849215ff6bef61 Oct 30 12:50:54.192051 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:50:54.191987 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0471989_f151_47a5_9d39_97d85760da9e.slice/crio-60a8edae6ab58dbde1749a7fe3300f27610a9f73b828c178f347fe8191447be0 WatchSource:0}: Error finding container 60a8edae6ab58dbde1749a7fe3300f27610a9f73b828c178f347fe8191447be0: Status 404 returned error can't find the container with id 60a8edae6ab58dbde1749a7fe3300f27610a9f73b828c178f347fe8191447be0 Oct 30 12:50:54.426720 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.426681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:54.426900 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.426815 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:50:54.426900 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.426882 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs podName:b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:55.426865731 +0000 UTC m=+4.005555295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs") pod "network-metrics-daemon-l7pxj" (UID: "b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:50:54.527037 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.527003 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:54.527217 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.527158 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:50:54.527292 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.527281 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret podName:00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:55.527260071 +0000 UTC m=+4.105949628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret") pod "global-pull-secret-syncer-jtc6z" (UID: "00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:50:54.627493 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.627465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:50:54.627644 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.627627 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:50:54.627720 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.627653 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:50:54.627720 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.627663 2569 projected.go:196] Error preparing data for projected volume kube-api-access-pm47z for pod openshift-network-diagnostics/network-check-target-b9v2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:50:54.627720 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:54.627714 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z podName:b6151a5c-6392-4151-8c74-6d518f34b4d0 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:55.627702685 +0000 UTC m=+4.206392229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pm47z" (UniqueName: "kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z") pod "network-check-target-b9v2t" (UID: "b6151a5c-6392-4151-8c74-6d518f34b4d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:50:54.861726 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.861610 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:45:53 +0000 UTC" deadline="2028-05-06 22:20:46.249035007 +0000 UTC" Oct 30 12:50:54.861726 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.861671 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13305h29m51.387368393s" Oct 30 12:50:54.884683 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.884652 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" event={"ID":"f0297b8f-b8e8-470e-a0cb-838697059ff4","Type":"ContainerStarted","Data":"3956a38b2219f01a26c99d1ea7cb75dfeaefd9601fc67f4398a1465d239d0062"} Oct 30 12:50:54.894835 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.894801 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" event={"ID":"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f","Type":"ContainerStarted","Data":"565530e1715a000908604d622d667bbca327e49f621cf4c1c0cd1bc52f64b7e0"} Oct 30 12:50:54.906701 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.904240 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s78dq" event={"ID":"15a59c67-94e0-48e5-b8ff-28a55dea0996","Type":"ContainerStarted","Data":"60edcebe872bf55f027cc442100417d57793ec9b989f58f1b541cdb7ecec3b2e"} Oct 30 12:50:54.907226 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.907195 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" event={"ID":"bac4afb01d6e1d9bf4827fcda4b8b291","Type":"ContainerStarted","Data":"ef0cc9de70cab4a13f0fc9952a6e508f66b556eb829d08381d8392749672726c"} Oct 30 12:50:54.915397 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.915295 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v4l8n" event={"ID":"d0471989-f151-47a5-9d39-97d85760da9e","Type":"ContainerStarted","Data":"60a8edae6ab58dbde1749a7fe3300f27610a9f73b828c178f347fe8191447be0"} Oct 30 12:50:54.920690 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.920604 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n4rct" event={"ID":"28a37fed-2af0-45d1-8e84-c658dbff6b61","Type":"ContainerStarted","Data":"cc486ad4df2adafbc04482c3cc486169273ab8751d5532c121849215ff6bef61"} Oct 30 12:50:54.925617 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.925550 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6p76v" event={"ID":"126669c1-ee47-49ab-a6f3-3592a66b1fd2","Type":"ContainerStarted","Data":"64a2cfedc1955d48656db6ae6abe61e0e481cc1c4f72783e9e5a30f386901db9"} Oct 30 12:50:54.931037 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.930938 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" event={"ID":"b2767673-ded2-4026-8b3f-ff2582c0aede","Type":"ContainerStarted","Data":"d29a281f0d2d0407f08ee5cda8a9ab237a8fa4ba655bc351fa5033dc58e47186"} Oct 30 12:50:54.937217 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.937154 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" event={"ID":"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3","Type":"ContainerStarted","Data":"bbc64b3d91231d34f5dc0e5857e20f292310dcd87c48b60ca0303964ef8f18de"} Oct 30 12:50:54.940422 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.940342 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cdptr" event={"ID":"b0aa96f3-47c7-48f1-98a3-dd3c4548d593","Type":"ContainerStarted","Data":"027a98a5d9752e6e47d1a3545787666832bc6f7d9dc2a465428d4c3e538577f9"} Oct 30 12:50:54.954347 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:54.954221 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" podStartSLOduration=1.954205545 podStartE2EDuration="1.954205545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:54.952159817 +0000 UTC m=+3.530849385" watchObservedRunningTime="2026-10-30 12:50:54.954205545 +0000 UTC m=+3.532895111" Oct 30 12:50:55.435851 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:55.435200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:55.435851 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.435362 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:50:55.435851 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.435420 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs podName:b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:57.435403334 +0000 UTC m=+6.014092881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs") pod "network-metrics-daemon-l7pxj" (UID: "b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:50:55.536818 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:55.536217 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:55.536818 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.536432 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:50:55.536818 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.536494 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret podName:00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:57.536477923 +0000 UTC m=+6.115167485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret") pod "global-pull-secret-syncer-jtc6z" (UID: "00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:50:55.638222 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:55.637678 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:50:55.638222 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.637820 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:50:55.638222 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.637841 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:50:55.638222 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.637854 2569 projected.go:196] Error preparing data for projected volume kube-api-access-pm47z for pod openshift-network-diagnostics/network-check-target-b9v2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:50:55.638222 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.637907 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z podName:b6151a5c-6392-4151-8c74-6d518f34b4d0 nodeName:}" failed. No retries permitted until 2026-10-30 12:50:57.637889504 +0000 UTC m=+6.216579061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pm47z" (UniqueName: "kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z") pod "network-check-target-b9v2t" (UID: "b6151a5c-6392-4151-8c74-6d518f34b4d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:50:55.862132 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:55.862058 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:45:53 +0000 UTC" deadline="2028-06-30 02:07:35.882720654 +0000 UTC" Oct 30 12:50:55.862132 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:55.862094 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14605h16m40.020632293s" Oct 30 12:50:55.873080 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:55.872432 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:50:55.873080 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.872546 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:50:55.873080 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:55.872928 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:55.873080 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.873027 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:50:55.873657 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:55.873469 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:55.873657 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:55.873579 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:50:55.957124 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:55.957087 2569 generic.go:356] "Generic (PLEG): container finished" podID="9a61b1589e9c4b5de6634f4c13443710" containerID="14b11276f940ddf79840d6409eabf15fc7383af2c179412f0e14608cde55e13e" exitCode=0 Oct 30 12:50:55.958145 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:55.958120 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" event={"ID":"9a61b1589e9c4b5de6634f4c13443710","Type":"ContainerDied","Data":"14b11276f940ddf79840d6409eabf15fc7383af2c179412f0e14608cde55e13e"} Oct 30 12:50:56.968855 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:56.968603 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" event={"ID":"9a61b1589e9c4b5de6634f4c13443710","Type":"ContainerStarted","Data":"4bb9e981a048a6712eaaf42af182b7e83e97955cd0f0fb66144c29e10d511f83"} Oct 30 12:50:57.011958 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:57.011169 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" podStartSLOduration=4.011150726 podStartE2EDuration="4.011150726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:57.01083846 +0000 UTC m=+5.589528018" watchObservedRunningTime="2026-10-30 12:50:57.011150726 +0000 UTC m=+5.589840295" Oct 30 12:50:57.457085 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:57.456980 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:57.457247 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.457145 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:50:57.457247 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.457207 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs podName:b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:01.45718551 +0000 UTC m=+10.035875067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs") pod "network-metrics-daemon-l7pxj" (UID: "b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:50:57.558160 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:57.558123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:57.558415 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.558299 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:50:57.558415 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.558368 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret podName:00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:01.558350638 +0000 UTC m=+10.137040194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret") pod "global-pull-secret-syncer-jtc6z" (UID: "00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:50:57.659770 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:57.659102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:50:57.659770 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.659295 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:50:57.659770 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.659313 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:50:57.659770 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.659339 2569 projected.go:196] Error preparing data for projected volume kube-api-access-pm47z for pod openshift-network-diagnostics/network-check-target-b9v2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:50:57.659770 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.659397 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z podName:b6151a5c-6392-4151-8c74-6d518f34b4d0 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:01.659378253 +0000 UTC m=+10.238067821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pm47z" (UniqueName: "kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z") pod "network-check-target-b9v2t" (UID: "b6151a5c-6392-4151-8c74-6d518f34b4d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:50:57.874308 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:57.874270 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:50:57.874453 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.874422 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:50:57.874812 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:57.874789 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:57.874903 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.874884 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:50:57.874968 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:57.874945 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:57.875047 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:57.875029 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:50:57.976903 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:57.976857 2569 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="539a633d742c27ad563e5d88c5cd07f85ecec9f20ecbdb9262a0281375720da9" exitCode=0 Oct 30 12:50:57.977912 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:57.977886 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" event={"ID":"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f","Type":"ContainerDied","Data":"539a633d742c27ad563e5d88c5cd07f85ecec9f20ecbdb9262a0281375720da9"} Oct 30 12:50:58.987936 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:58.987901 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cdptr" event={"ID":"b0aa96f3-47c7-48f1-98a3-dd3c4548d593","Type":"ContainerStarted","Data":"e26a716b07e9e56523204d36200fc37479dd46a7253f20392ae19f805314f1b8"} Oct 30 12:50:59.874267 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:59.874229 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:50:59.874453 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:59.874379 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:50:59.874786 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:59.874764 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:50:59.874883 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:59.874863 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:50:59.874939 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:59.874933 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:50:59.875024 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:50:59.875008 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:50:59.997756 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:50:59.997699 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6p76v" event={"ID":"126669c1-ee47-49ab-a6f3-3592a66b1fd2","Type":"ContainerStarted","Data":"f350b97b651eb7095eea7fc955e6bebcbc0575b30a1661157d8c0ac9b16d86e3"} Oct 30 12:51:00.044071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:00.043029 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cdptr" podStartSLOduration=4.298431181 podStartE2EDuration="8.04301117s" totalImagesPullingTime="3.744579989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:59.028126786 +0000 UTC m=+7.606816351" watchObservedRunningTime="2026-10-30 12:51:00.04301117 +0000 UTC m=+8.621700735" Oct 30 12:51:01.002486 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:01.002391 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" event={"ID":"f0297b8f-b8e8-470e-a0cb-838697059ff4","Type":"ContainerStarted","Data":"0c58727ac51b1b025db3a4dde1b085de56dda167543a4b8751aeb6fd95c10d51"} Oct 30 12:51:01.495073 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:01.495039 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:01.495305 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.495229 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:01.495305 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.495290 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs podName:b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:09.495277143 +0000 UTC m=+18.073966698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs") pod "network-metrics-daemon-l7pxj" (UID: "b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:01.505528 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:01.505487 2569 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:51:01.595782 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:01.595748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:01.595947 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.595919 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:01.596034 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.595976 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret podName:00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:09.595961332 +0000 UTC m=+18.174650875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret") pod "global-pull-secret-syncer-jtc6z" (UID: "00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:01.696552 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:01.696512 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:01.696728 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.696716 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:01.696791 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.696736 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:01.696791 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.696749 2569 projected.go:196] Error preparing data for projected volume kube-api-access-pm47z for pod openshift-network-diagnostics/network-check-target-b9v2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:01.696888 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.696808 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z podName:b6151a5c-6392-4151-8c74-6d518f34b4d0 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:09.696789866 +0000 UTC m=+18.275479428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pm47z" (UniqueName: "kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z") pod "network-check-target-b9v2t" (UID: "b6151a5c-6392-4151-8c74-6d518f34b4d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:01.874631 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:01.874608 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:01.874740 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.874715 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:01.874785 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:01.874773 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:01.874882 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.874841 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:01.874916 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:01.874884 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:01.874973 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:01.874957 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:02.019777 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.019629 2569 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:51:01.505504947Z","UUID":"bee11c44-d5da-4119-aa89-b3494cef2aaf","Handler":null,"Name":"","Endpoint":""} Oct 30 12:51:02.020849 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.020823 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s78dq" event={"ID":"15a59c67-94e0-48e5-b8ff-28a55dea0996","Type":"ContainerStarted","Data":"74477c425a4418110dda29dda49b2d4bb9559761e8f31b67e42432cba51451af"} Oct 30 12:51:02.025487 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.022594 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:51:02.025487 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.022634 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:51:02.026752 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.025418 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v4l8n" event={"ID":"d0471989-f151-47a5-9d39-97d85760da9e","Type":"ContainerStarted","Data":"934019babf6bfcab16d376ce7b02a695dd59da7ac97bb6e3bb025d328b092be8"} Oct 30 12:51:02.031094 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.031061 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" event={"ID":"f0297b8f-b8e8-470e-a0cb-838697059ff4","Type":"ContainerStarted","Data":"e9c6aa09d748bb0981538630ee2034a8c4e1027b3d9e00aca2c597433baed439"} Oct 30 12:51:02.068982 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.068925 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6p76v" podStartSLOduration=5.283036607 podStartE2EDuration="10.068905243s" totalImagesPullingTime="4.785868636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:00.043979671 +0000 UTC m=+8.622669237" watchObservedRunningTime="2026-10-30 12:51:02.068905243 +0000 UTC m=+10.647594808" Oct 30 12:51:02.110056 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.109967 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-s78dq" podStartSLOduration=2.419058027 podStartE2EDuration="10.109947821s" totalImagesPullingTime="7.690889794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:02.06803856 +0000 UTC m=+10.646728126" watchObservedRunningTime="2026-10-30 12:51:02.109947821 +0000 UTC m=+10.688637387" Oct 30 12:51:02.110334 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.110289 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-v4l8n" podStartSLOduration=2.7252530139999998 podStartE2EDuration="10.110282125s" totalImagesPullingTime="7.385029111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:02.110063006 +0000 UTC m=+10.688752553" watchObservedRunningTime="2026-10-30 12:51:02.110282125 +0000 UTC m=+10.688971689" Oct 30 12:51:02.288054 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.287860 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:51:02.288627 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:02.288604 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:51:03.035877 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:03.035785 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" event={"ID":"f0297b8f-b8e8-470e-a0cb-838697059ff4","Type":"ContainerStarted","Data":"b0649df216e3e92bff6a5078f779719046874466c196aa44660159c51ffa0826"} Oct 30 12:51:03.037805 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:03.037776 2569 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="9140e24858863571c58a832430da31d9c76e14ff3158e04cc4b3f231aabf42c2" exitCode=0 Oct 30 12:51:03.037899 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:03.037879 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" event={"ID":"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f","Type":"ContainerDied","Data":"9140e24858863571c58a832430da31d9c76e14ff3158e04cc4b3f231aabf42c2"} Oct 30 12:51:03.077290 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:03.077247 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" podStartSLOduration=3.069653684 podStartE2EDuration="11.077230426s" totalImagesPullingTime="8.007576742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:03.077218736 +0000 UTC m=+11.655908301" watchObservedRunningTime="2026-10-30 12:51:03.077230426 +0000 UTC m=+11.655919985" Oct 30 12:51:03.871912 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:03.871745 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:03.872031 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:03.871800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:03.872031 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:03.872000 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:03.872140 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:03.871839 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:03.872140 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:03.872094 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:03.872207 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:03.872153 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:04.042074 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:04.041999 2569 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="b6009e3868ada8cde78324dec93155308834a4dbd706e0abf43e8f308fb8aa57" exitCode=0 Oct 30 12:51:04.042462 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:04.042084 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" event={"ID":"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f","Type":"ContainerDied","Data":"b6009e3868ada8cde78324dec93155308834a4dbd706e0abf43e8f308fb8aa57"} Oct 30 12:51:04.044024 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:04.043994 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n4rct" event={"ID":"28a37fed-2af0-45d1-8e84-c658dbff6b61","Type":"ContainerStarted","Data":"b79c7f10bfd9b64228ba9a5de34978aded6429fc42c273a67aef4153bf5c8500"} Oct 30 12:51:04.045707 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:04.045668 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" event={"ID":"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3","Type":"ContainerStarted","Data":"03e9f44cb1fcd1195729520cd2d4c21ca0d233905ad840938ff6f90d4aab7898"} Oct 30 12:51:04.126453 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:04.126407 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n4rct" podStartSLOduration=4.389883501 podStartE2EDuration="12.126395052s" totalImagesPullingTime="7.736511551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:04.126226118 +0000 UTC m=+12.704915684" watchObservedRunningTime="2026-10-30 12:51:04.126395052 +0000 UTC m=+12.705084614" Oct 30 12:51:04.176820 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:04.176764 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" podStartSLOduration=2.640282821 podStartE2EDuration="12.17675085s" totalImagesPullingTime="9.536468029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:04.175864419 +0000 UTC m=+12.754553986" watchObservedRunningTime="2026-10-30 12:51:04.17675085 +0000 UTC m=+12.755440416" Oct 30 12:51:05.631375 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:05.631337 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:51:05.631979 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:05.631490 2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:51:05.631979 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:05.631918 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:51:05.871629 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:05.871581 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:05.871796 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:05.871585 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:05.871796 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:05.871733 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:05.871796 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:05.871585 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:05.871936 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:05.871807 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:05.871936 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:05.871863 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:06.052851 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:06.052814 2569 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="63ff6221fb4cfdaf242f32a24d6a5b84c46ad668f3c21c523b023c9bf1fd8ce5" exitCode=0 Oct 30 12:51:06.053013 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:06.052900 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" event={"ID":"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f","Type":"ContainerDied","Data":"63ff6221fb4cfdaf242f32a24d6a5b84c46ad668f3c21c523b023c9bf1fd8ce5"} Oct 30 12:51:07.871190 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:07.871160 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:07.871190 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:07.871187 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:07.871776 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:07.871160 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:07.871776 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:07.871265 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:07.871776 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:07.871357 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:07.871776 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:07.871404 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:08.059546 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:08.059516 2569 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="01852d67e6b4283f31054162adfd1b413b8c698973022763f6b9236bb2e0078b" exitCode=0 Oct 30 12:51:08.059686 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:08.059562 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" event={"ID":"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f","Type":"ContainerDied","Data":"01852d67e6b4283f31054162adfd1b413b8c698973022763f6b9236bb2e0078b"} Oct 30 12:51:09.064442 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.064083 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" event={"ID":"b2767673-ded2-4026-8b3f-ff2582c0aede","Type":"ContainerStarted","Data":"defeefdb9558748195ca6ce30de0ba9f8f942fb8c45095eeb6fda55913702bef"} Oct 30 12:51:09.064881 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.064517 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" event={"ID":"b2767673-ded2-4026-8b3f-ff2582c0aede","Type":"ContainerStarted","Data":"a0a9fbd9c16bd8ad0d6ac53ae30d9068449a24a0275d1a1ee56d29cf545306db"} Oct 30 12:51:09.064881 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.064557 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" event={"ID":"b2767673-ded2-4026-8b3f-ff2582c0aede","Type":"ContainerStarted","Data":"51d953528097805390724b28b6590821c4e9682e8f1167749423f1cbd69afb1a"} Oct 30 12:51:09.064881 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.064588 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" event={"ID":"b2767673-ded2-4026-8b3f-ff2582c0aede","Type":"ContainerStarted","Data":"fe0635d7a52ae69b9cf8a49bae635c72b960810f44fff50a76a9b8c96f6a0d18"} Oct 30 12:51:09.064881 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.064633 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" event={"ID":"b2767673-ded2-4026-8b3f-ff2582c0aede","Type":"ContainerStarted","Data":"6abcc30d281d5c03ba4b4ec20808b78fe1ac914d045b3fe324f5186a2caa8375"} Oct 30 12:51:09.064881 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.064679 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" event={"ID":"b2767673-ded2-4026-8b3f-ff2582c0aede","Type":"ContainerStarted","Data":"1d96c0367be0a6137665cebd5d22b68ad0f8b8cf294b56dee5f9925868a49b17"} Oct 30 12:51:09.066938 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.066917 2569 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="ab5addeb3b8ed7ad31567b27d99a41aa611606a9ec73d29ddfa8e52fdb19c7af" exitCode=0 Oct 30 12:51:09.067018 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.066952 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" event={"ID":"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f","Type":"ContainerDied","Data":"ab5addeb3b8ed7ad31567b27d99a41aa611606a9ec73d29ddfa8e52fdb19c7af"} Oct 30 12:51:09.563039 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.563008 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:09.563199 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.563108 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:09.563199 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.563155 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs podName:b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.563143047 +0000 UTC m=+34.141832590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs") pod "network-metrics-daemon-l7pxj" (UID: "b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:09.663298 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.663262 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:09.663488 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.663415 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:09.663488 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.663476 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret podName:00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.663460684 +0000 UTC m=+34.242150227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret") pod "global-pull-secret-syncer-jtc6z" (UID: "00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:09.764415 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.764388 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:09.764606 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.764496 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:09.764606 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.764510 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:09.764606 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.764518 2569 projected.go:196] Error preparing data for projected volume kube-api-access-pm47z for pod openshift-network-diagnostics/network-check-target-b9v2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:09.764606 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.764562 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z podName:b6151a5c-6392-4151-8c74-6d518f34b4d0 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.76454964 +0000 UTC m=+34.343239182 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pm47z" (UniqueName: "kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z") pod "network-check-target-b9v2t" (UID: "b6151a5c-6392-4151-8c74-6d518f34b4d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:09.876833 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.876761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:09.876833 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.876785 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:09.876996 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:09.876770 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:09.876996 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.876874 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:09.876996 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.876926 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:09.877087 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:09.877049 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:10.071741 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:10.071703 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" event={"ID":"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f","Type":"ContainerStarted","Data":"8f9f26b631751e4318961b6f3ee778a171c3ddcf39f01381e19556ce963d22e1"} Oct 30 12:51:10.131768 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:10.131694 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dnwpp" podStartSLOduration=7.335775958 podStartE2EDuration="18.131681286s" totalImagesPullingTime="10.795905328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:10.130882095 +0000 UTC m=+18.709571660" watchObservedRunningTime="2026-10-30 12:51:10.131681286 +0000 UTC m=+18.710370850" Oct 30 12:51:11.077689 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:11.077653 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" event={"ID":"b2767673-ded2-4026-8b3f-ff2582c0aede","Type":"ContainerStarted","Data":"3e9416ca4d403273ca1147d884d3ed45b354143e07e29252da6d5ff444e5f892"} Oct 30 12:51:11.871983 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:11.871954 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:11.872144 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:11.872035 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:11.872144 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:11.872112 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:11.872260 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:11.872243 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:11.872295 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:11.872284 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:11.872363 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:11.872351 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:13.874254 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:13.874222 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:13.874254 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:13.874251 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:13.874995 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:13.874357 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:13.874995 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:13.874384 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:13.874995 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:13.874441 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:13.874995 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:13.874506 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:14.089046 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:14.089012 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" event={"ID":"b2767673-ded2-4026-8b3f-ff2582c0aede","Type":"ContainerStarted","Data":"103a90599ba3e4991f33255944a2a51b9e563fc99843c78d57d0afd3d8b4fd80"} Oct 30 12:51:14.089287 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:14.089265 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:14.089387 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:14.089352 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:14.104757 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:14.104724 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:14.209924 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:14.209833 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" podStartSLOduration=7.74289766 podStartE2EDuration="22.20982046s" totalImagesPullingTime="14.4669228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:14.209504058 +0000 UTC m=+22.788193624" watchObservedRunningTime="2026-10-30 12:51:14.20982046 +0000 UTC m=+22.788510022" Oct 30 12:51:15.091691 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:15.091428 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:15.105944 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:15.105921 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:15.238887 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:15.238855 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b9v2t"] Oct 30 12:51:15.239055 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:15.238972 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:15.239093 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:15.239066 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:15.241513 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:15.241479 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jtc6z"] Oct 30 12:51:15.241648 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:15.241587 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:15.241705 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:15.241682 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:15.242142 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:15.242119 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l7pxj"] Oct 30 12:51:15.242231 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:15.242213 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:15.242316 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:15.242299 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:16.871358 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:16.871308 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:16.871358 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:16.871349 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:16.871840 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:16.871353 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:16.871840 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:16.871439 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:16.871840 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:16.871495 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:16.871840 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:16.871582 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:18.871561 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:18.871524 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:18.871561 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:18.871555 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:18.872059 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:18.871639 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:18.872059 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:18.871697 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:18.872059 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:18.871775 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:18.872059 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:18.871855 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:20.872004 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:20.871966 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:20.872004 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:20.871988 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:20.872457 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:20.871966 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:20.872457 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:20.872091 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7pxj" podUID="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" Oct 30 12:51:20.872457 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:20.872152 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtc6z" podUID="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" Oct 30 12:51:20.872457 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:20.872204 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9v2t" podUID="b6151a5c-6392-4151-8c74-6d518f34b4d0" Oct 30 12:51:21.727780 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.727753 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeReady" Oct 30 12:51:21.727936 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.727877 2569 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 12:51:21.788009 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.787972 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-crmm8"] Oct 30 12:51:21.796254 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.796220 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-97qv2"] Oct 30 12:51:21.796410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.796369 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-crmm8" Oct 30 12:51:21.799675 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.799651 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bqzs4"] Oct 30 12:51:21.799806 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.799788 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-97qv2" Oct 30 12:51:21.802795 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.802626 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw"] Oct 30 12:51:21.802892 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.802801 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bqzs4" Oct 30 12:51:21.811949 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.811927 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l2xkr"] Oct 30 12:51:21.812098 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.812078 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw" Oct 30 12:51:21.812617 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.812597 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.814832 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.814812 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm"] Oct 30 12:51:21.814926 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.814899 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:21.817585 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.817568 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6qdnl"] Oct 30 12:51:21.818558 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.818543 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:21.819862 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.819844 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85cb969f47-rcssw"] Oct 30 12:51:21.819983 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.819965 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:21.823148 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.822894 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-df858cf77-mnl8f"] Oct 30 12:51:21.823148 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.823031 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:21.826847 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.826828 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-87dcb74f4-755j7"] Oct 30 12:51:21.827003 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.826979 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:21.833347 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.831942 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh"] Oct 30 12:51:21.833347 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.832042 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:21.837048 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.837031 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd"] Oct 30 12:51:21.837180 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.837163 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:21.840157 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.840140 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn"] Oct 30 12:51:21.840273 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.840259 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd" Oct 30 12:51:21.842631 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.842617 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-f4hf7"] Oct 30 12:51:21.842749 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.842734 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:21.844383 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.844365 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.845729 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.845672 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv"] Oct 30 12:51:21.845873 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.845843 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:21.848533 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.848518 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-97qv2"] Oct 30 12:51:21.848617 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.848544 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pzn2g"] Oct 30 12:51:21.848654 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.848636 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:21.851433 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.851415 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-crmm8"] Oct 30 12:51:21.851497 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.851447 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-94rws"] Oct 30 12:51:21.851497 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.851477 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:21.854969 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.854952 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bqzs4"] Oct 30 12:51:21.855046 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.854983 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l2xkr"] Oct 30 12:51:21.855081 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.855051 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94rws" Oct 30 12:51:21.856253 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.856235 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 12:51:21.856525 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.856510 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-z66zl\"" Oct 30 12:51:21.856597 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.856529 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mt9m7\"" Oct 30 12:51:21.859710 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.859682 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm"] Oct 30 12:51:21.862268 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.862248 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd"] Oct 30 12:51:21.862459 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.862444 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-df858cf77-mnl8f"] Oct 30 12:51:21.868588 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.868571 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh"] Oct 30 12:51:21.870579 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.870564 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6qdnl"] Oct 30 12:51:21.884534 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.884512 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw"] Oct 30 12:51:21.896599 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.895729 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv"] Oct 30 12:51:21.896790 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.896772 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85cb969f47-rcssw"] Oct 30 12:51:21.899871 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.899555 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94rws"] Oct 30 12:51:21.904254 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.904230 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn"] Oct 30 12:51:21.906480 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.906458 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-f4hf7"] Oct 30 12:51:21.906594 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.906507 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-87dcb74f4-755j7"] Oct 30 12:51:21.914407 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.913670 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.914407 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.913971 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.914407 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.914166 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 12:51:21.914407 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.914208 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.914407 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.914344 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.914407 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.914386 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.914715 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.914472 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 12:51:21.914715 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.914614 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:51:21.915984 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.915968 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.916158 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916142 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.916216 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916190 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 12:51:21.916292 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916278 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.916368 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916289 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.916368 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916293 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.916368 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916364 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.916481 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916293 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 12:51:21.916481 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916377 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.916481 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916398 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 12:51:21.916481 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916410 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 12:51:21.916481 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.916413 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.925620 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.925370 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 12:51:21.925620 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.925370 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.925620 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.925405 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.925620 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.925428 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.927937 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.927921 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 12:51:21.928023 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.927951 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:51:21.928023 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.927967 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.928401 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.928371 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 12:51:21.938238 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.938216 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 12:51:21.939837 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.939822 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.940025 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.940012 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:51:21.940160 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.940147 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 12:51:21.941659 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.941643 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pzn2g"] Oct 30 12:51:21.947394 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:21.947442 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:21.947442 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947430 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-certificates\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:21.947535 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947481 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-image-registry-private-configuration\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:21.947593 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947550 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-image-registry-private-configuration\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:21.947638 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-serving-cert\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:21.947668 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caed0908-537e-4064-858e-f3f8a82b87b1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:21.947709 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:21.947755 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcdfs\" (UniqueName: \"kubernetes.io/projected/116675da-08c0-4101-835c-48193661ccbf-kube-api-access-dcdfs\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:21.947821 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/332bd965-1f4d-4749-8922-946a21e7ce75-ca-trust-extracted\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:21.947863 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947852 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116675da-08c0-4101-835c-48193661ccbf-tmp-dir\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:21.947905 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947876 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:21.947905 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947894 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:21.947980 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947910 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc2w4\" (UniqueName: \"kubernetes.io/projected/cbb88cf5-e088-4260-82e6-ed56a2b00176-kube-api-access-pc2w4\") pod \"kas-connection-checker-685895c7c9-crmm8\" (UID: \"cbb88cf5-e088-4260-82e6-ed56a2b00176\") " pod="kube-system/kas-connection-checker-685895c7c9-crmm8" Oct 30 12:51:21.947980 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947929 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-ca-trust-extracted\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:21.948057 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.947980 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6ww\" (UniqueName: \"kubernetes.io/projected/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-kube-api-access-ck6ww\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:21.948057 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948017 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-trusted-ca\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:21.948159 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948073 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqh6\" (UniqueName: \"kubernetes.io/projected/0cbb909f-28ac-4b1e-9a95-5b8870886b61-kube-api-access-mpqh6\") pod \"ingress-canary-pzn2g\" (UID: \"0cbb909f-28ac-4b1e-9a95-5b8870886b61\") " pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:21.948159 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948109 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:21.948159 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948138 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-installation-pull-secrets\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:21.948298 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948163 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-default-certificate\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:21.948298 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948193 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cbb909f-28ac-4b1e-9a95-5b8870886b61-cert\") pod \"ingress-canary-pzn2g\" (UID: \"0cbb909f-28ac-4b1e-9a95-5b8870886b61\") " pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:21.948298 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948216 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-registry-certificates\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:21.948298 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxxk\" (UniqueName: \"kubernetes.io/projected/35925210-93d9-4a34-b733-6481da8921f3-kube-api-access-npxxk\") pod \"kas-connection-checker-685895c7c9-97qv2\" (UID: \"35925210-93d9-4a34-b733-6481da8921f3\") " pod="kube-system/kas-connection-checker-685895c7c9-97qv2" Oct 30 12:51:21.948298 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948291 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ml9\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-kube-api-access-s6ml9\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:21.948550 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948355 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/116675da-08c0-4101-835c-48193661ccbf-config-volume\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:21.948550 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948387 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-service-ca-bundle\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:21.948550 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948459 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:21.948550 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948487 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-bound-sa-token\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:21.948550 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948510 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:21.948729 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:21.948729 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948582 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndg29\" (UniqueName: \"kubernetes.io/projected/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-kube-api-access-ndg29\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:21.948729 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948602 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-trusted-ca\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:21.948729 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948624 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-bound-sa-token\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:21.948729 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948646 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkp4n\" (UniqueName: \"kubernetes.io/projected/9dc9e5e5-426c-4e0b-897f-54e0e28d3313-kube-api-access-vkp4n\") pod \"volume-data-source-validator-58c9f5f46d-vf8sw\" (UID: \"9dc9e5e5-426c-4e0b-897f-54e0e28d3313\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw" Oct 30 12:51:21.948729 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948661 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqflv\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-kube-api-access-jqflv\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:21.948729 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948675 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-tmp\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:21.948729 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948696 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1fa5e9-0704-4469-83b4-bdb65f799855-serving-cert\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:21.949071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948766 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7x4b\" (UniqueName: \"kubernetes.io/projected/6e6304cb-862e-43ea-8688-8298f61b8c5b-kube-api-access-x7x4b\") pod \"network-check-source-6f78d59f99-68gnd\" (UID: \"6e6304cb-862e-43ea-8688-8298f61b8c5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd" Oct 30 12:51:21.949071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948811 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-snapshots\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:21.949071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948859 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264f3a6d-7988-4ee8-bf15-bf13e991a63c-serving-cert\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:21.949071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948884 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g99qq\" (UniqueName: \"kubernetes.io/projected/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-kube-api-access-g99qq\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:21.949071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq4nn\" (UniqueName: \"kubernetes.io/projected/173ce15b-35dd-4a98-8f29-d13c62b56206-kube-api-access-sq4nn\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:21.949071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.948949 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvsp\" (UniqueName: \"kubernetes.io/projected/3400463b-e945-48d9-8dc5-8b8106cb6e26-kube-api-access-8lvsp\") pod \"kas-connection-checker-685895c7c9-bqzs4\" (UID: \"3400463b-e945-48d9-8dc5-8b8106cb6e26\") " pod="kube-system/kas-connection-checker-685895c7c9-bqzs4" Oct 30 12:51:21.949071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949015 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-stats-auth\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:21.949071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949047 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:21.949071 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:21.949356 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949094 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:21.949356 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949124 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-config\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:21.949356 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949206 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-config\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:21.949356 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949221 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-trusted-ca\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:21.949356 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949239 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xfl\" (UniqueName: \"kubernetes.io/projected/264f3a6d-7988-4ee8-bf15-bf13e991a63c-kube-api-access-j8xfl\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:21.949356 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949256 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-installation-pull-secrets\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:21.949356 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949271 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmn4\" (UniqueName: \"kubernetes.io/projected/fc1fa5e9-0704-4469-83b4-bdb65f799855-kube-api-access-gsmn4\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:21.949356 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949293 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:21.949356 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.949316 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7m2h\" (UniqueName: \"kubernetes.io/projected/482cc1b2-387f-4b1e-98f5-adada095ffec-kube-api-access-c7m2h\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:21.951551 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.951533 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 12:51:21.952171 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.952155 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 12:51:21.952357 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.952343 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 12:51:21.952437 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.952425 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 12:51:21.952756 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.952744 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:51:21.953394 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953360 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 12:51:21.953394 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953376 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 12:51:21.953394 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953387 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 12:51:21.953546 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953392 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wb7zr\"" Oct 30 12:51:21.953728 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953712 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 12:51:21.953728 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953721 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:51:21.953865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953725 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 12:51:21.953865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953725 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 12:51:21.953865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953809 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5jsfn\"" Oct 30 12:51:21.953865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953820 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 12:51:21.953865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953841 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h5w6k\"" Oct 30 12:51:21.954041 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953879 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4vf72\"" Oct 30 12:51:21.954041 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953888 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rq622\"" Oct 30 12:51:21.954041 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953898 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhkwq\"" Oct 30 12:51:21.954041 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953903 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-24gr7\"" Oct 30 12:51:21.954041 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.953934 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 12:51:21.954041 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.954002 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r9ldj\"" Oct 30 12:51:21.954193 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:21.954058 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vr75\"" Oct 30 12:51:22.050316 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050292 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:22.050479 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:22.050479 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-certificates\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.050479 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050377 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-image-registry-private-configuration\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.050479 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-image-registry-private-configuration\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.050479 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050408 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-serving-cert\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:22.050479 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050435 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caed0908-537e-4064-858e-f3f8a82b87b1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:22.050479 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.050452 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.050521 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls podName:fcd977c1-bd5b-466b-a9bd-aa4889ed86b4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:22.550502173 +0000 UTC m=+31.129191737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlskm" (UID: "fcd977c1-bd5b-466b-a9bd-aa4889ed86b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.050540 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:22.55053143 +0000 UTC m=+31.129220976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : configmap references non-existent config key: service-ca.crt Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050456 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcdfs\" (UniqueName: \"kubernetes.io/projected/116675da-08c0-4101-835c-48193661ccbf-kube-api-access-dcdfs\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050605 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/332bd965-1f4d-4749-8922-946a21e7ce75-ca-trust-extracted\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050634 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116675da-08c0-4101-835c-48193661ccbf-tmp-dir\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050658 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050708 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc2w4\" (UniqueName: \"kubernetes.io/projected/cbb88cf5-e088-4260-82e6-ed56a2b00176-kube-api-access-pc2w4\") pod \"kas-connection-checker-685895c7c9-crmm8\" (UID: \"cbb88cf5-e088-4260-82e6-ed56a2b00176\") " pod="kube-system/kas-connection-checker-685895c7c9-crmm8" Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-ca-trust-extracted\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050747 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6ww\" (UniqueName: \"kubernetes.io/projected/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-kube-api-access-ck6ww\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-trusted-ca\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.050803 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqh6\" (UniqueName: \"kubernetes.io/projected/0cbb909f-28ac-4b1e-9a95-5b8870886b61-kube-api-access-mpqh6\") pod \"ingress-canary-pzn2g\" (UID: \"0cbb909f-28ac-4b1e-9a95-5b8870886b61\") " pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050848 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-installation-pull-secrets\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050896 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-default-certificate\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cbb909f-28ac-4b1e-9a95-5b8870886b61-cert\") pod \"ingress-canary-pzn2g\" (UID: \"0cbb909f-28ac-4b1e-9a95-5b8870886b61\") " pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-registry-certificates\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050966 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxxk\" (UniqueName: \"kubernetes.io/projected/35925210-93d9-4a34-b733-6481da8921f3-kube-api-access-npxxk\") pod \"kas-connection-checker-685895c7c9-97qv2\" (UID: \"35925210-93d9-4a34-b733-6481da8921f3\") " pod="kube-system/kas-connection-checker-685895c7c9-97qv2" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.050991 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ml9\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-kube-api-access-s6ml9\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051005 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116675da-08c0-4101-835c-48193661ccbf-tmp-dir\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051014 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/116675da-08c0-4101-835c-48193661ccbf-config-volume\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051037 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-service-ca-bundle\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051043 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-certificates\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051115 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-bound-sa-token\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:22.051410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndg29\" (UniqueName: \"kubernetes.io/projected/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-kube-api-access-ndg29\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-trusted-ca\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-bound-sa-token\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051266 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkp4n\" (UniqueName: \"kubernetes.io/projected/9dc9e5e5-426c-4e0b-897f-54e0e28d3313-kube-api-access-vkp4n\") pod \"volume-data-source-validator-58c9f5f46d-vf8sw\" (UID: \"9dc9e5e5-426c-4e0b-897f-54e0e28d3313\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051287 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqflv\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-kube-api-access-jqflv\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051306 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-tmp\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1fa5e9-0704-4469-83b4-bdb65f799855-serving-cert\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7x4b\" (UniqueName: \"kubernetes.io/projected/6e6304cb-862e-43ea-8688-8298f61b8c5b-kube-api-access-x7x4b\") pod \"network-check-source-6f78d59f99-68gnd\" (UID: \"6e6304cb-862e-43ea-8688-8298f61b8c5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-snapshots\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051423 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264f3a6d-7988-4ee8-bf15-bf13e991a63c-serving-cert\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051450 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g99qq\" (UniqueName: \"kubernetes.io/projected/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-kube-api-access-g99qq\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051474 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq4nn\" (UniqueName: \"kubernetes.io/projected/173ce15b-35dd-4a98-8f29-d13c62b56206-kube-api-access-sq4nn\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051497 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvsp\" (UniqueName: \"kubernetes.io/projected/3400463b-e945-48d9-8dc5-8b8106cb6e26-kube-api-access-8lvsp\") pod \"kas-connection-checker-685895c7c9-bqzs4\" (UID: \"3400463b-e945-48d9-8dc5-8b8106cb6e26\") " pod="kube-system/kas-connection-checker-685895c7c9-bqzs4" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-stats-auth\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051557 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/332bd965-1f4d-4749-8922-946a21e7ce75-ca-trust-extracted\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.052107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051655 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-config\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051711 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-config\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051744 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-trusted-ca\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xfl\" (UniqueName: \"kubernetes.io/projected/264f3a6d-7988-4ee8-bf15-bf13e991a63c-kube-api-access-j8xfl\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-installation-pull-secrets\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.051145 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051882 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmn4\" (UniqueName: \"kubernetes.io/projected/fc1fa5e9-0704-4469-83b4-bdb65f799855-kube-api-access-gsmn4\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.051913 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls podName:116675da-08c0-4101-835c-48193661ccbf nodeName:}" failed. No retries permitted until 2026-10-30 12:51:22.551898685 +0000 UTC m=+31.130588242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls") pod "dns-default-94rws" (UID: "116675da-08c0-4101-835c-48193661ccbf") : secret "dns-default-metrics-tls" not found Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051933 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051960 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7m2h\" (UniqueName: \"kubernetes.io/projected/482cc1b2-387f-4b1e-98f5-adada095ffec-kube-api-access-c7m2h\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.051974 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.052064 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-trusted-ca\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.052348 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.052403 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert podName:caed0908-537e-4064-858e-f3f8a82b87b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:22.55238919 +0000 UTC m=+31.131078747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-f4hf7" (UID: "caed0908-537e-4064-858e-f3f8a82b87b1") : secret "networking-console-plugin-cert" not found Oct 30 12:51:22.052865 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.052540 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-snapshots\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.053660 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.052589 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:22.053660 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.052819 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:22.053660 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.051643 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:51:22.053660 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.052953 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87dcb74f4-755j7: secret "image-registry-tls" not found Oct 30 12:51:22.053660 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.051864 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:51:22.053660 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.052968 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cb969f47-rcssw: secret "image-registry-tls" not found Oct 30 12:51:22.053660 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.053565 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-trusted-ca\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:22.053999 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.053664 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-config\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:22.054202 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.054168 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:22.054785 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.054755 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-config\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:22.054912 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.054886 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:22.055047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.055027 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caed0908-537e-4064-858e-f3f8a82b87b1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:22.055116 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.055095 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls podName:1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:22.555080585 +0000 UTC m=+31.133770150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls") pod "image-registry-87dcb74f4-755j7" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88") : secret "image-registry-tls" not found Oct 30 12:51:22.055185 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.055172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-ca-trust-extracted\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.055239 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.055212 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-service-ca-bundle\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.055443 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.055426 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls podName:332bd965-1f4d-4749-8922-946a21e7ce75 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:22.555406758 +0000 UTC m=+31.134096316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls") pod "image-registry-85cb969f47-rcssw" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75") : secret "image-registry-tls" not found Oct 30 12:51:22.055550 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.055527 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:51:22.055676 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.055664 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:22.555647991 +0000 UTC m=+31.134337552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : secret "router-metrics-certs-default" not found Oct 30 12:51:22.055761 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.055585 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:51:22.056058 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.055825 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-trusted-ca\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.056058 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.055855 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls podName:482cc1b2-387f-4b1e-98f5-adada095ffec nodeName:}" failed. No retries permitted until 2026-10-30 12:51:22.555842676 +0000 UTC m=+31.134532235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jsqvh" (UID: "482cc1b2-387f-4b1e-98f5-adada095ffec") : secret "samples-operator-tls" not found Oct 30 12:51:22.056058 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.055974 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/116675da-08c0-4101-835c-48193661ccbf-config-volume\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:22.056246 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.056073 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-serving-cert\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:22.056246 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.056138 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-image-registry-private-configuration\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.056246 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.056154 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264f3a6d-7988-4ee8-bf15-bf13e991a63c-serving-cert\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:22.056417 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.056308 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-registry-certificates\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.056417 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.056358 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-installation-pull-secrets\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.056594 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.056571 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-tmp\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.057232 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.057212 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-image-registry-private-configuration\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.057691 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.057673 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cbb909f-28ac-4b1e-9a95-5b8870886b61-cert\") pod \"ingress-canary-pzn2g\" (UID: \"0cbb909f-28ac-4b1e-9a95-5b8870886b61\") " pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:22.058183 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.058165 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-installation-pull-secrets\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.058610 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.058590 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-stats-auth\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:22.058734 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.058717 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1fa5e9-0704-4469-83b4-bdb65f799855-serving-cert\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.059103 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.059080 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-default-certificate\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:22.082616 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.082592 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq4nn\" (UniqueName: \"kubernetes.io/projected/173ce15b-35dd-4a98-8f29-d13c62b56206-kube-api-access-sq4nn\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:22.083238 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.083215 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-bound-sa-token\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.083433 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.083412 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xfl\" (UniqueName: \"kubernetes.io/projected/264f3a6d-7988-4ee8-bf15-bf13e991a63c-kube-api-access-j8xfl\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:22.084334 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.084302 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqh6\" (UniqueName: \"kubernetes.io/projected/0cbb909f-28ac-4b1e-9a95-5b8870886b61-kube-api-access-mpqh6\") pod \"ingress-canary-pzn2g\" (UID: \"0cbb909f-28ac-4b1e-9a95-5b8870886b61\") " pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:22.085685 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.085664 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvsp\" (UniqueName: \"kubernetes.io/projected/3400463b-e945-48d9-8dc5-8b8106cb6e26-kube-api-access-8lvsp\") pod \"kas-connection-checker-685895c7c9-bqzs4\" (UID: \"3400463b-e945-48d9-8dc5-8b8106cb6e26\") " pod="kube-system/kas-connection-checker-685895c7c9-bqzs4" Oct 30 12:51:22.088146 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.088123 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6ww\" (UniqueName: \"kubernetes.io/projected/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-kube-api-access-ck6ww\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:22.088332 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.088297 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7m2h\" (UniqueName: \"kubernetes.io/projected/482cc1b2-387f-4b1e-98f5-adada095ffec-kube-api-access-c7m2h\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:22.088854 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.088833 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc2w4\" (UniqueName: \"kubernetes.io/projected/cbb88cf5-e088-4260-82e6-ed56a2b00176-kube-api-access-pc2w4\") pod \"kas-connection-checker-685895c7c9-crmm8\" (UID: \"cbb88cf5-e088-4260-82e6-ed56a2b00176\") " pod="kube-system/kas-connection-checker-685895c7c9-crmm8" Oct 30 12:51:22.089855 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.089838 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmn4\" (UniqueName: \"kubernetes.io/projected/fc1fa5e9-0704-4469-83b4-bdb65f799855-kube-api-access-gsmn4\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.090075 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.090058 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxxk\" (UniqueName: \"kubernetes.io/projected/35925210-93d9-4a34-b733-6481da8921f3-kube-api-access-npxxk\") pod \"kas-connection-checker-685895c7c9-97qv2\" (UID: \"35925210-93d9-4a34-b733-6481da8921f3\") " pod="kube-system/kas-connection-checker-685895c7c9-97qv2" Oct 30 12:51:22.095913 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.095892 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-bound-sa-token\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.096184 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.096168 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndg29\" (UniqueName: \"kubernetes.io/projected/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-kube-api-access-ndg29\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:22.096841 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.096822 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7x4b\" (UniqueName: \"kubernetes.io/projected/6e6304cb-862e-43ea-8688-8298f61b8c5b-kube-api-access-x7x4b\") pod \"network-check-source-6f78d59f99-68gnd\" (UID: \"6e6304cb-862e-43ea-8688-8298f61b8c5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd" Oct 30 12:51:22.097122 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.097099 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g99qq\" (UniqueName: \"kubernetes.io/projected/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-kube-api-access-g99qq\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:22.097260 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.097241 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqflv\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-kube-api-access-jqflv\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.099371 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.099354 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcdfs\" (UniqueName: \"kubernetes.io/projected/116675da-08c0-4101-835c-48193661ccbf-kube-api-access-dcdfs\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:22.104872 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.104851 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkp4n\" (UniqueName: \"kubernetes.io/projected/9dc9e5e5-426c-4e0b-897f-54e0e28d3313-kube-api-access-vkp4n\") pod \"volume-data-source-validator-58c9f5f46d-vf8sw\" (UID: \"9dc9e5e5-426c-4e0b-897f-54e0e28d3313\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw" Oct 30 12:51:22.105093 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.105073 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ml9\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-kube-api-access-s6ml9\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.113944 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.113924 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-crmm8" Oct 30 12:51:22.122947 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.122926 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-97qv2" Oct 30 12:51:22.130066 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.130046 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bqzs4" Oct 30 12:51:22.139037 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.139016 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw" Oct 30 12:51:22.146180 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.146163 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:22.158358 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.158317 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:22.204412 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.204383 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd" Oct 30 12:51:22.211711 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.211682 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:22.225620 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.225576 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:22.230816 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.230794 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:22.528354 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.528309 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-crmm8"] Oct 30 12:51:22.529952 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:51:22.529917 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb88cf5_e088_4260_82e6_ed56a2b00176.slice/crio-efe28e4e8d6e998580d6116017ab7ce7ff749ca4595eb34f4089e8e6b4aa2ec7 WatchSource:0}: Error finding container efe28e4e8d6e998580d6116017ab7ce7ff749ca4595eb34f4089e8e6b4aa2ec7: Status 404 returned error can't find the container with id efe28e4e8d6e998580d6116017ab7ce7ff749ca4595eb34f4089e8e6b4aa2ec7 Oct 30 12:51:22.531478 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.531462 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6qdnl"] Oct 30 12:51:22.531861 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:51:22.531842 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1fa5e9_0704_4469_83b4_bdb65f799855.slice/crio-e0c544b2eaa4ce3ca8a3893f688d1004913ced0c0c32744fd6874c8a4f520dc0 WatchSource:0}: Error finding container e0c544b2eaa4ce3ca8a3893f688d1004913ced0c0c32744fd6874c8a4f520dc0: Status 404 returned error can't find the container with id e0c544b2eaa4ce3ca8a3893f688d1004913ced0c0c32744fd6874c8a4f520dc0 Oct 30 12:51:22.538770 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.538750 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-97qv2"] Oct 30 12:51:22.560546 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.560486 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:22.560646 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.560539 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:22.560646 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.560580 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:22.560646 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.560629 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:22.560789 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.560655 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:22.560789 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560658 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:51:22.560789 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.560704 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:22.560789 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560717 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:51:22.560789 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560727 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert podName:caed0908-537e-4064-858e-f3f8a82b87b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:23.56070813 +0000 UTC m=+32.139397687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-f4hf7" (UID: "caed0908-537e-4064-858e-f3f8a82b87b1") : secret "networking-console-plugin-cert" not found Oct 30 12:51:22.560789 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560663 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:22.560789 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560771 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:23.56075504 +0000 UTC m=+32.139444604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : secret "router-metrics-certs-default" not found Oct 30 12:51:22.560789 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560779 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:51:22.560789 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560786 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:51:22.561128 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560816 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87dcb74f4-755j7: secret "image-registry-tls" not found Oct 30 12:51:22.561128 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560794 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls podName:116675da-08c0-4101-835c-48193661ccbf nodeName:}" failed. No retries permitted until 2026-10-30 12:51:23.560784813 +0000 UTC m=+32.139474361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls") pod "dns-default-94rws" (UID: "116675da-08c0-4101-835c-48193661ccbf") : secret "dns-default-metrics-tls" not found Oct 30 12:51:22.561128 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560890 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls podName:482cc1b2-387f-4b1e-98f5-adada095ffec nodeName:}" failed. No retries permitted until 2026-10-30 12:51:23.560866152 +0000 UTC m=+32.139555701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jsqvh" (UID: "482cc1b2-387f-4b1e-98f5-adada095ffec") : secret "samples-operator-tls" not found Oct 30 12:51:22.561128 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.560914 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls podName:1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:23.560904554 +0000 UTC m=+32.139594102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls") pod "image-registry-87dcb74f4-755j7" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88") : secret "image-registry-tls" not found Oct 30 12:51:22.561128 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.560964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:22.561128 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.561006 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:22.561128 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.561091 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:22.561128 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.561112 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:23.561100722 +0000 UTC m=+32.139790269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : configmap references non-existent config key: service-ca.crt Oct 30 12:51:22.561468 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.561132 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls podName:fcd977c1-bd5b-466b-a9bd-aa4889ed86b4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:23.561120584 +0000 UTC m=+32.139810140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlskm" (UID: "fcd977c1-bd5b-466b-a9bd-aa4889ed86b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:22.561468 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.561187 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:51:22.561468 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.561200 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cb969f47-rcssw: secret "image-registry-tls" not found Oct 30 12:51:22.561468 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:22.561244 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls podName:332bd965-1f4d-4749-8922-946a21e7ce75 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:23.561230836 +0000 UTC m=+32.139920381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls") pod "image-registry-85cb969f47-rcssw" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75") : secret "image-registry-tls" not found Oct 30 12:51:22.565052 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.564964 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw"] Oct 30 12:51:22.567944 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.567926 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bqzs4"] Oct 30 12:51:22.653096 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.652441 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l2xkr"] Oct 30 12:51:22.656252 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.656227 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd"] Oct 30 12:51:22.660638 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.660614 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pzn2g"] Oct 30 12:51:22.663128 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:51:22.663018 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbb909f_28ac_4b1e_9a95_5b8870886b61.slice/crio-b665cb8d66f7fa16481c292520994f406684d6ae84ce19bf7271a5396956cf25 WatchSource:0}: Error finding container b665cb8d66f7fa16481c292520994f406684d6ae84ce19bf7271a5396956cf25: Status 404 returned error can't find the container with id b665cb8d66f7fa16481c292520994f406684d6ae84ce19bf7271a5396956cf25 Oct 30 12:51:22.675046 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.675025 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn"] Oct 30 12:51:22.676627 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:51:22.676604 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a06a4b4_ab53_4ba8_803c_fd2fe71bd9fe.slice/crio-250d94911962dbd46d73c952d9be812d6870122389f57106b8d8a3d4bad7eac6 WatchSource:0}: Error finding container 250d94911962dbd46d73c952d9be812d6870122389f57106b8d8a3d4bad7eac6: Status 404 returned error can't find the container with id 250d94911962dbd46d73c952d9be812d6870122389f57106b8d8a3d4bad7eac6 Oct 30 12:51:22.714066 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.714036 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv"] Oct 30 12:51:22.719176 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:51:22.719155 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a8078f_bd8e_42d8_86ef_4e0030e0ea76.slice/crio-4ca4804dd5facaee14638f9bffe0814faa9af1cd905a1d446a36a59b183a1920 WatchSource:0}: Error finding container 4ca4804dd5facaee14638f9bffe0814faa9af1cd905a1d446a36a59b183a1920: Status 404 returned error can't find the container with id 4ca4804dd5facaee14638f9bffe0814faa9af1cd905a1d446a36a59b183a1920 Oct 30 12:51:22.871904 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.871821 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:22.871904 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.871862 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:22.871904 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.871864 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:22.892594 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.892578 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:51:22.893169 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.893147 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:51:22.893169 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.893167 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n54s8\"" Oct 30 12:51:22.893330 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.893180 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rnjq7\"" Oct 30 12:51:22.893330 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:22.893230 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb6s2\"" Oct 30 12:51:23.110052 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.110009 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd" event={"ID":"6e6304cb-862e-43ea-8688-8298f61b8c5b","Type":"ContainerStarted","Data":"76b60094adf3e50acd5ec3de283723592c4ad82d118f03a96a6c93ef5152f110"} Oct 30 12:51:23.111378 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.111346 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw" event={"ID":"9dc9e5e5-426c-4e0b-897f-54e0e28d3313","Type":"ContainerStarted","Data":"12f9bb1918bc92524b935d14cd072890cddf9faf0646ad13663cbe23fcf18d10"} Oct 30 12:51:23.112868 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.112843 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bqzs4" event={"ID":"3400463b-e945-48d9-8dc5-8b8106cb6e26","Type":"ContainerStarted","Data":"3041f9c116dfa9be2b52de4f3fa88f426ae49ca7a9ad2af4b8c50bc8a9b68fdb"} Oct 30 12:51:23.112959 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.112888 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bqzs4" event={"ID":"3400463b-e945-48d9-8dc5-8b8106cb6e26","Type":"ContainerStarted","Data":"da0500037ffeac56eb34a48abc809696866cfc8cf4463187dcc189cce7bd4659"} Oct 30 12:51:23.114048 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.114024 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" event={"ID":"56a8078f-bd8e-42d8-86ef-4e0030e0ea76","Type":"ContainerStarted","Data":"4ca4804dd5facaee14638f9bffe0814faa9af1cd905a1d446a36a59b183a1920"} Oct 30 12:51:23.115505 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.115477 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-crmm8" event={"ID":"cbb88cf5-e088-4260-82e6-ed56a2b00176","Type":"ContainerStarted","Data":"da5900c52c96620ad9a13c701414e0770c3a30a3cc6c1e49d7612b4d1a211971"} Oct 30 12:51:23.115609 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.115516 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-crmm8" event={"ID":"cbb88cf5-e088-4260-82e6-ed56a2b00176","Type":"ContainerStarted","Data":"efe28e4e8d6e998580d6116017ab7ce7ff749ca4595eb34f4089e8e6b4aa2ec7"} Oct 30 12:51:23.116578 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.116555 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" event={"ID":"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe","Type":"ContainerStarted","Data":"250d94911962dbd46d73c952d9be812d6870122389f57106b8d8a3d4bad7eac6"} Oct 30 12:51:23.117789 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.117762 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pzn2g" event={"ID":"0cbb909f-28ac-4b1e-9a95-5b8870886b61","Type":"ContainerStarted","Data":"b665cb8d66f7fa16481c292520994f406684d6ae84ce19bf7271a5396956cf25"} Oct 30 12:51:23.118958 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.118933 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" event={"ID":"264f3a6d-7988-4ee8-bf15-bf13e991a63c","Type":"ContainerStarted","Data":"fcc75f970f6ed346b1d2485c1135a8ff4551b9c0e2b9cbc7af01538a61b9779f"} Oct 30 12:51:23.120184 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.120140 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6qdnl" event={"ID":"fc1fa5e9-0704-4469-83b4-bdb65f799855","Type":"ContainerStarted","Data":"e0c544b2eaa4ce3ca8a3893f688d1004913ced0c0c32744fd6874c8a4f520dc0"} Oct 30 12:51:23.121654 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.121629 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-97qv2" event={"ID":"35925210-93d9-4a34-b733-6481da8921f3","Type":"ContainerStarted","Data":"687d00acc2affe96a1b03ba72eaaeefc775bbf866fa2559a34cc5967176fbd3f"} Oct 30 12:51:23.121751 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.121671 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-97qv2" event={"ID":"35925210-93d9-4a34-b733-6481da8921f3","Type":"ContainerStarted","Data":"129bffca37c4391ace11af0211c2545e03d673fd42f09996673efc220baa21d8"} Oct 30 12:51:23.158858 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.158754 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bqzs4" podStartSLOduration=371.158736811 podStartE2EDuration="6m11.158736811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:23.158559949 +0000 UTC m=+31.737249513" watchObservedRunningTime="2026-10-30 12:51:23.158736811 +0000 UTC m=+31.737426379" Oct 30 12:51:23.201271 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.201214 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-crmm8" podStartSLOduration=371.201196117 podStartE2EDuration="6m11.201196117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:23.201019289 +0000 UTC m=+31.779708855" watchObservedRunningTime="2026-10-30 12:51:23.201196117 +0000 UTC m=+31.779885696" Oct 30 12:51:23.242491 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.242437 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-97qv2" podStartSLOduration=371.242419985 podStartE2EDuration="6m11.242419985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:23.241462222 +0000 UTC m=+31.820151788" watchObservedRunningTime="2026-10-30 12:51:23.242419985 +0000 UTC m=+31.821109551" Oct 30 12:51:23.571709 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.571507 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:23.571709 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.571582 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:23.571709 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.571611 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:23.571709 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.571662 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:23.571709 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.571688 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:23.572072 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.571745 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:23.572072 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.571793 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:23.572072 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:23.571833 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:23.572072 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572000 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.571982488 +0000 UTC m=+34.150672045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : configmap references non-existent config key: service-ca.crt Oct 30 12:51:23.572267 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572099 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:51:23.572267 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572136 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert podName:caed0908-537e-4064-858e-f3f8a82b87b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.572125493 +0000 UTC m=+34.150815050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-f4hf7" (UID: "caed0908-537e-4064-858e-f3f8a82b87b1") : secret "networking-console-plugin-cert" not found Oct 30 12:51:23.572267 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572196 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:23.572267 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572225 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls podName:116675da-08c0-4101-835c-48193661ccbf nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.572217489 +0000 UTC m=+34.150907048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls") pod "dns-default-94rws" (UID: "116675da-08c0-4101-835c-48193661ccbf") : secret "dns-default-metrics-tls" not found Oct 30 12:51:23.572490 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572284 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:51:23.572490 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572295 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cb969f47-rcssw: secret "image-registry-tls" not found Oct 30 12:51:23.572490 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572481 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:51:23.572624 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572562 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:51:23.572669 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572629 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:51:23.572669 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572638 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87dcb74f4-755j7: secret "image-registry-tls" not found Oct 30 12:51:23.572859 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.572717 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:23.573108 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.573093 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls podName:332bd965-1f4d-4749-8922-946a21e7ce75 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.57231457 +0000 UTC m=+34.151004126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls") pod "image-registry-85cb969f47-rcssw" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75") : secret "image-registry-tls" not found Oct 30 12:51:23.573168 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.573130 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls podName:482cc1b2-387f-4b1e-98f5-adada095ffec nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.57311894 +0000 UTC m=+34.151808497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jsqvh" (UID: "482cc1b2-387f-4b1e-98f5-adada095ffec") : secret "samples-operator-tls" not found Oct 30 12:51:23.573236 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.573169 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.573136891 +0000 UTC m=+34.151826439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : secret "router-metrics-certs-default" not found Oct 30 12:51:23.573236 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.573185 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls podName:1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.573176681 +0000 UTC m=+34.151866628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls") pod "image-registry-87dcb74f4-755j7" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88") : secret "image-registry-tls" not found Oct 30 12:51:23.573236 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:23.573202 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls podName:fcd977c1-bd5b-466b-a9bd-aa4889ed86b4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:25.573191755 +0000 UTC m=+34.151881306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlskm" (UID: "fcd977c1-bd5b-466b-a9bd-aa4889ed86b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:24.612941 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:24.612862 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:24.860532 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:24.860453 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:25.600337 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.600287 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:25.600511 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.600348 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:25.600511 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.600389 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:25.600511 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.600425 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:25.600511 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.600480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:25.600710 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.600517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:25.600710 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.600545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:25.600710 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.600599 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:25.600710 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.600629 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:25.600882 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.600768 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:51:25.600882 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.600784 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87dcb74f4-755j7: secret "image-registry-tls" not found Oct 30 12:51:25.600882 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.600841 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls podName:1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.600823094 +0000 UTC m=+38.179512655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls") pod "image-registry-87dcb74f4-755j7" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88") : secret "image-registry-tls" not found Oct 30 12:51:25.601024 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.600977 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:51:25.601024 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.600994 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:51:25.601109 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601043 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.601026751 +0000 UTC m=+38.179716309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : secret "router-metrics-certs-default" not found Oct 30 12:51:25.601109 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601066 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:51:25.601109 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601077 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cb969f47-rcssw: secret "image-registry-tls" not found Oct 30 12:51:25.601271 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601145 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 12:51:25.601271 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601159 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:25.601271 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601066 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert podName:caed0908-537e-4064-858e-f3f8a82b87b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.601055362 +0000 UTC m=+38.179744918 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-f4hf7" (UID: "caed0908-537e-4064-858e-f3f8a82b87b1") : secret "networking-console-plugin-cert" not found Oct 30 12:51:25.601271 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601219 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls podName:332bd965-1f4d-4749-8922-946a21e7ce75 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.60120006 +0000 UTC m=+38.179889618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls") pod "image-registry-85cb969f47-rcssw" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75") : secret "image-registry-tls" not found Oct 30 12:51:25.601271 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601230 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:51:25.601271 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601237 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs podName:b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:57.601227686 +0000 UTC m=+66.179917235 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs") pod "network-metrics-daemon-l7pxj" (UID: "b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3") : secret "metrics-daemon-secret" not found Oct 30 12:51:25.601271 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601258 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls podName:fcd977c1-bd5b-466b-a9bd-aa4889ed86b4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.601248237 +0000 UTC m=+38.179937788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlskm" (UID: "fcd977c1-bd5b-466b-a9bd-aa4889ed86b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:25.601271 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601273 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.601265727 +0000 UTC m=+38.179955276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : configmap references non-existent config key: service-ca.crt Oct 30 12:51:25.601689 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601293 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls podName:482cc1b2-387f-4b1e-98f5-adada095ffec nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.601285354 +0000 UTC m=+38.179974908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jsqvh" (UID: "482cc1b2-387f-4b1e-98f5-adada095ffec") : secret "samples-operator-tls" not found Oct 30 12:51:25.601689 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601302 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:25.601689 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:25.601354 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls podName:116675da-08c0-4101-835c-48193661ccbf nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.601340966 +0000 UTC m=+38.180030523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls") pod "dns-default-94rws" (UID: "116675da-08c0-4101-835c-48193661ccbf") : secret "dns-default-metrics-tls" not found Oct 30 12:51:25.701883 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.701840 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:25.722974 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.722917 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:25.802787 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.802746 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:25.810107 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.809996 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:25.900392 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.900091 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:25.905410 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:25.905353 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:26.138676 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:26.138624 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw" event={"ID":"9dc9e5e5-426c-4e0b-897f-54e0e28d3313","Type":"ContainerStarted","Data":"b6e14dabed6f4159dd0a4489be2b99e57fd53ca79f2ee39061fe770e2bd01037"} Oct 30 12:51:26.178875 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:26.178760 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw" podStartSLOduration=328.724196402 podStartE2EDuration="5m31.178743129s" totalImagesPullingTime="2.454546727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:26.178614304 +0000 UTC m=+34.757303869" watchObservedRunningTime="2026-10-30 12:51:26.178743129 +0000 UTC m=+34.757432696" Oct 30 12:51:26.258379 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:26.257397 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b9v2t"] Oct 30 12:51:26.291007 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:26.290932 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jtc6z"] Oct 30 12:51:26.367809 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:26.367776 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:26.553771 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:26.553729 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:27.145966 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:27.145932 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jtc6z" event={"ID":"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1","Type":"ContainerStarted","Data":"d24bd95c7d6564ae3fe6506b5002a9f05323f62aa5abd3f4526c8f0ddf5a45ae"} Oct 30 12:51:27.147885 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:27.147862 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6qdnl" event={"ID":"fc1fa5e9-0704-4469-83b4-bdb65f799855","Type":"ContainerStarted","Data":"09f4d59355af2847938c07260800ecdcaf5161be90c95419c2f8e80a39f0abe9"} Oct 30 12:51:27.150019 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:27.149999 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b9v2t" event={"ID":"b6151a5c-6392-4151-8c74-6d518f34b4d0","Type":"ContainerStarted","Data":"65256a7c5313aee03415a40f1bae0a0424acc69602780bc70d87be25fdb5911c"} Oct 30 12:51:27.193282 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:27.193221 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-6qdnl" podStartSLOduration=327.712271087 podStartE2EDuration="5m32.193203039s" totalImagesPullingTime="4.480931952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:27.192160151 +0000 UTC m=+35.770849717" watchObservedRunningTime="2026-10-30 12:51:27.193203039 +0000 UTC m=+35.771892605" Oct 30 12:51:28.157824 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.157779 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" event={"ID":"56a8078f-bd8e-42d8-86ef-4e0030e0ea76","Type":"ContainerStarted","Data":"4954ad3fecc2927d789fb6c5cb56c23d831c868a7e622bd8cf29caf09ae1a000"} Oct 30 12:51:28.161955 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.161553 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" event={"ID":"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe","Type":"ContainerStarted","Data":"e3495df0605543f40ae65970bc49fe5b9f7e2ecf598307287a7ff003b92f8dc5"} Oct 30 12:51:28.165531 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.165504 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" event={"ID":"264f3a6d-7988-4ee8-bf15-bf13e991a63c","Type":"ContainerStarted","Data":"bd7b99f9ffe1380bb0b0c5216b5b2526ee2ed13cad791c7b19be44cd656bb4d9"} Oct 30 12:51:28.165644 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.165552 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:28.178771 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.178742 2569 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-l2xkr container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" start-of-body= Oct 30 12:51:28.178883 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.178795 2569 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" podUID="264f3a6d-7988-4ee8-bf15-bf13e991a63c" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" Oct 30 12:51:28.204799 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.204745 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" podStartSLOduration=327.081111002 podStartE2EDuration="5m32.204727488s" totalImagesPullingTime="5.123616486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:28.203405428 +0000 UTC m=+36.782094992" watchObservedRunningTime="2026-10-30 12:51:28.204727488 +0000 UTC m=+36.783417057" Oct 30 12:51:28.248999 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.248918 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" podStartSLOduration=328.044474794 podStartE2EDuration="5m33.248901452s" totalImagesPullingTime="5.204426658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:28.247531939 +0000 UTC m=+36.826221508" watchObservedRunningTime="2026-10-30 12:51:28.248901452 +0000 UTC m=+36.827591018" Oct 30 12:51:28.262348 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.261448 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bzlvd"] Oct 30 12:51:28.280280 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.279301 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.300471 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.297618 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" podStartSLOduration=327.112507672 podStartE2EDuration="5m32.297604825s" totalImagesPullingTime="5.185097153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:28.297355283 +0000 UTC m=+36.876044850" watchObservedRunningTime="2026-10-30 12:51:28.297604825 +0000 UTC m=+36.876294409" Oct 30 12:51:28.301229 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.300943 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:51:28.301982 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.301901 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzlvd"] Oct 30 12:51:28.303441 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.303348 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:51:28.304301 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.304281 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7mcfh-fnbft"] Oct 30 12:51:28.308155 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.308117 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kq2jq\"" Oct 30 12:51:28.340214 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.340116 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7mcfh-fnbft"] Oct 30 12:51:28.340305 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.340286 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.354220 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.354145 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:28.430984 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.430780 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58b3cad0-f9c1-4547-bb19-f1fd20041eca-data-volume\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.430984 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.430843 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-service-ca-bundle\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.430984 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.430941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.431232 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.431073 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.431232 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.431109 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f13020b-562d-470d-9bc4-70a5dd26f1e9-archives-path\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.431232 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.431208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58b3cad0-f9c1-4547-bb19-f1fd20041eca-crio-socket\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.431397 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.431236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv69b\" (UniqueName: \"kubernetes.io/projected/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-api-access-qv69b\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.431397 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.431286 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k2f6\" (UniqueName: \"kubernetes.io/projected/4f13020b-562d-470d-9bc4-70a5dd26f1e9-kube-api-access-4k2f6\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.431397 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.431359 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-trusted-ca-bundle\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.520754 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.520664 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:28.533011 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.532979 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.533183 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533041 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.533183 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533063 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f13020b-562d-470d-9bc4-70a5dd26f1e9-archives-path\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.533183 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533097 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58b3cad0-f9c1-4547-bb19-f1fd20041eca-crio-socket\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.533183 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533114 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv69b\" (UniqueName: \"kubernetes.io/projected/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-api-access-qv69b\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.533183 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k2f6\" (UniqueName: \"kubernetes.io/projected/4f13020b-562d-470d-9bc4-70a5dd26f1e9-kube-api-access-4k2f6\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.533183 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533167 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-trusted-ca-bundle\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.533477 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533211 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58b3cad0-f9c1-4547-bb19-f1fd20041eca-data-volume\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.533477 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533237 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-service-ca-bundle\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.534287 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:28.533664 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:51:28.534287 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:28.533737 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.033718129 +0000 UTC m=+37.612407688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : configmap "kube-rbac-proxy" not found Oct 30 12:51:28.534287 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533825 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-service-ca-bundle\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.534287 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:28.533907 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:28.534287 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:28.533937 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.033928864 +0000 UTC m=+37.612618431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:28.534287 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533977 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58b3cad0-f9c1-4547-bb19-f1fd20041eca-crio-socket\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.534287 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.533978 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f13020b-562d-470d-9bc4-70a5dd26f1e9-archives-path\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.534287 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.534233 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58b3cad0-f9c1-4547-bb19-f1fd20041eca-data-volume\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.534724 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.534605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-trusted-ca-bundle\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.559733 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.559674 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k2f6\" (UniqueName: \"kubernetes.io/projected/4f13020b-562d-470d-9bc4-70a5dd26f1e9-kube-api-access-4k2f6\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:28.561254 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.561232 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv69b\" (UniqueName: \"kubernetes.io/projected/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-api-access-qv69b\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:28.720468 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:28.720394 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:29.040272 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.040184 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:29.040448 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.040419 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:29.040672 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.040559 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:29.040672 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.040571 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:51:29.040672 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.040629 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:30.040610772 +0000 UTC m=+38.619300315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : configmap "kube-rbac-proxy" not found Oct 30 12:51:29.040672 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.040644 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:30.04063722 +0000 UTC m=+38.619326764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:29.112900 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.112865 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:29.154799 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.154279 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7mcfh-fnbft"] Oct 30 12:51:29.169908 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.169877 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd" event={"ID":"6e6304cb-862e-43ea-8688-8298f61b8c5b","Type":"ContainerStarted","Data":"8d2f774e4a4332c0e99f01456a82595e3fecfd0ceca3b61ac4a9f9c0c0ec4f85"} Oct 30 12:51:29.172406 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.172378 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pzn2g" event={"ID":"0cbb909f-28ac-4b1e-9a95-5b8870886b61","Type":"ContainerStarted","Data":"77b61c20407fcde8832d27d0285a8c36f4d2b78380bf45d30f5929548085de4d"} Oct 30 12:51:29.174311 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.174289 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/0.log" Oct 30 12:51:29.174420 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.174345 2569 generic.go:356] "Generic (PLEG): container finished" podID="264f3a6d-7988-4ee8-bf15-bf13e991a63c" containerID="bd7b99f9ffe1380bb0b0c5216b5b2526ee2ed13cad791c7b19be44cd656bb4d9" exitCode=255 Oct 30 12:51:29.174489 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.174423 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" event={"ID":"264f3a6d-7988-4ee8-bf15-bf13e991a63c","Type":"ContainerDied","Data":"bd7b99f9ffe1380bb0b0c5216b5b2526ee2ed13cad791c7b19be44cd656bb4d9"} Oct 30 12:51:29.174731 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.174704 2569 scope.go:122] "RemoveContainer" containerID="bd7b99f9ffe1380bb0b0c5216b5b2526ee2ed13cad791c7b19be44cd656bb4d9" Oct 30 12:51:29.176480 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.176450 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7mcfh-fnbft" event={"ID":"4f13020b-562d-470d-9bc4-70a5dd26f1e9","Type":"ContainerStarted","Data":"ef12761b54de310bb0455cd4b2c0354e74c304518a7bdb116398633ff40056ad"} Oct 30 12:51:29.267627 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.267557 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd" podStartSLOduration=328.075758204 podStartE2EDuration="5m34.267538256s" totalImagesPullingTime="6.191780052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:29.26669757 +0000 UTC m=+37.845387135" watchObservedRunningTime="2026-10-30 12:51:29.267538256 +0000 UTC m=+37.846227824" Oct 30 12:51:29.416354 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.415640 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pzn2g" podStartSLOduration=2.833018715 podStartE2EDuration="8.415621588s" totalImagesPullingTime="5.582602873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:29.324232701 +0000 UTC m=+37.902922267" watchObservedRunningTime="2026-10-30 12:51:29.415621588 +0000 UTC m=+37.994311154" Oct 30 12:51:29.432054 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.432012 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:29.435194 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.435167 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv"] Oct 30 12:51:29.459400 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.459370 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww"] Oct 30 12:51:29.459587 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.459500 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv" Oct 30 12:51:29.475600 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.475560 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv"] Oct 30 12:51:29.475753 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.475724 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww" Oct 30 12:51:29.481036 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.481009 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4g9fn\"" Oct 30 12:51:29.494986 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.494920 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww"] Oct 30 12:51:29.564785 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.564733 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:29.564952 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.564867 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 12:51:29.646921 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.646893 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcdx\" (UniqueName: \"kubernetes.io/projected/da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac-kube-api-access-tvcdx\") pod \"migrator-66df5d496b-jntww\" (UID: \"da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww" Oct 30 12:51:29.647081 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.646944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjfn2\" (UniqueName: \"kubernetes.io/projected/8ebe4466-834b-4748-9fa5-6a01043ae3ad-kube-api-access-kjfn2\") pod \"migrator-66df5d496b-6dhsv\" (UID: \"8ebe4466-834b-4748-9fa5-6a01043ae3ad\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv" Oct 30 12:51:29.647081 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.647001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:29.647081 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.647030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:29.647081 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.647074 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:29.647287 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.647126 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:29.647287 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.647157 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:29.647287 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.647193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:29.647287 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.647226 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:29.647287 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.647254 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:29.647541 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.647392 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:51:29.647541 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.647406 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85cb969f47-rcssw: secret "image-registry-tls" not found Oct 30 12:51:29.647541 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.647452 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls podName:332bd965-1f4d-4749-8922-946a21e7ce75 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.647439838 +0000 UTC m=+46.226129393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls") pod "image-registry-85cb969f47-rcssw" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75") : secret "image-registry-tls" not found Oct 30 12:51:29.647841 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.647821 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:51:29.647923 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.647872 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.647858079 +0000 UTC m=+46.226547636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : secret "router-metrics-certs-default" not found Oct 30 12:51:29.647981 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.647952 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 12:51:29.647981 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.647965 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87dcb74f4-755j7: secret "image-registry-tls" not found Oct 30 12:51:29.648079 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.647991 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls podName:1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.647983289 +0000 UTC m=+46.226672845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls") pod "image-registry-87dcb74f4-755j7" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88") : secret "image-registry-tls" not found Oct 30 12:51:29.648079 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.648033 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:51:29.648079 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.648054 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls podName:482cc1b2-387f-4b1e-98f5-adada095ffec nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.648048751 +0000 UTC m=+46.226738307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jsqvh" (UID: "482cc1b2-387f-4b1e-98f5-adada095ffec") : secret "samples-operator-tls" not found Oct 30 12:51:29.648209 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.648087 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:29.648209 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.648105 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls podName:fcd977c1-bd5b-466b-a9bd-aa4889ed86b4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.648100046 +0000 UTC m=+46.226789602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlskm" (UID: "fcd977c1-bd5b-466b-a9bd-aa4889ed86b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:29.648209 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.648145 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.648139654 +0000 UTC m=+46.226829210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : configmap references non-existent config key: service-ca.crt Oct 30 12:51:29.648209 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.648178 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 12:51:29.648209 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.648198 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert podName:caed0908-537e-4064-858e-f3f8a82b87b1 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.648192761 +0000 UTC m=+46.226882317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-f4hf7" (UID: "caed0908-537e-4064-858e-f3f8a82b87b1") : secret "networking-console-plugin-cert" not found Oct 30 12:51:29.648396 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.648230 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:29.648396 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:29.648245 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls podName:116675da-08c0-4101-835c-48193661ccbf nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.648240196 +0000 UTC m=+46.226929752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls") pod "dns-default-94rws" (UID: "116675da-08c0-4101-835c-48193661ccbf") : secret "dns-default-metrics-tls" not found Oct 30 12:51:29.678294 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.678255 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:29.748455 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.748422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcdx\" (UniqueName: \"kubernetes.io/projected/da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac-kube-api-access-tvcdx\") pod \"migrator-66df5d496b-jntww\" (UID: \"da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww" Oct 30 12:51:29.748647 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.748470 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfn2\" (UniqueName: \"kubernetes.io/projected/8ebe4466-834b-4748-9fa5-6a01043ae3ad-kube-api-access-kjfn2\") pod \"migrator-66df5d496b-6dhsv\" (UID: \"8ebe4466-834b-4748-9fa5-6a01043ae3ad\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv" Oct 30 12:51:29.782157 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.782095 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfn2\" (UniqueName: \"kubernetes.io/projected/8ebe4466-834b-4748-9fa5-6a01043ae3ad-kube-api-access-kjfn2\") pod \"migrator-66df5d496b-6dhsv\" (UID: \"8ebe4466-834b-4748-9fa5-6a01043ae3ad\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv" Oct 30 12:51:29.782344 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.782303 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcdx\" (UniqueName: \"kubernetes.io/projected/da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac-kube-api-access-tvcdx\") pod \"migrator-66df5d496b-jntww\" (UID: \"da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww" Oct 30 12:51:29.792164 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.792136 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv" Oct 30 12:51:29.811721 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.811691 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww" Oct 30 12:51:29.845673 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.845641 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xzkxq"] Oct 30 12:51:29.890047 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.887297 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:29.893891 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.893860 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xzkxq"] Oct 30 12:51:29.907058 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.907001 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-42xpg\"" Oct 30 12:51:29.907447 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.907307 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 12:51:29.907447 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.907407 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 12:51:29.907603 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.907471 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 12:51:29.907603 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.907538 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 12:51:29.918891 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:29.918871 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:30.051621 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.051582 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-key\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:30.051810 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.051649 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-cabundle\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:30.051878 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.051804 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2m8\" (UniqueName: \"kubernetes.io/projected/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-kube-api-access-dm2m8\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:30.051966 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.051941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:30.052011 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.051998 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:30.052092 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:30.052073 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:30.052141 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:30.052101 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:51:30.052177 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:30.052149 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:32.052131123 +0000 UTC m=+40.630820673 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:30.052177 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:30.052170 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:32.052158663 +0000 UTC m=+40.630848215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : configmap "kube-rbac-proxy" not found Oct 30 12:51:30.117408 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.117313 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv"] Oct 30 12:51:30.118868 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:51:30.118841 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebe4466_834b_4748_9fa5_6a01043ae3ad.slice/crio-9f96260c566a49fbd2b3050c8111f1c6fba63c44108c4e6d1ebb0c8391d3f12f WatchSource:0}: Error finding container 9f96260c566a49fbd2b3050c8111f1c6fba63c44108c4e6d1ebb0c8391d3f12f: Status 404 returned error can't find the container with id 9f96260c566a49fbd2b3050c8111f1c6fba63c44108c4e6d1ebb0c8391d3f12f Oct 30 12:51:30.151715 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.151687 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww"] Oct 30 12:51:30.153247 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.153216 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2m8\" (UniqueName: \"kubernetes.io/projected/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-kube-api-access-dm2m8\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:30.153478 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.153459 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-key\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:30.153568 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.153509 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-cabundle\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:30.153716 ip-10-0-133-61 kubenswrapper[2569]: W1030 12:51:30.153694 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9ebb2a_bb7e_4fbd_b944_b2347cfff6ac.slice/crio-f18c301930b64494b7e5fb40bd01304e0d7f4942803d56b63f06f74ced418f47 WatchSource:0}: Error finding container f18c301930b64494b7e5fb40bd01304e0d7f4942803d56b63f06f74ced418f47: Status 404 returned error can't find the container with id f18c301930b64494b7e5fb40bd01304e0d7f4942803d56b63f06f74ced418f47 Oct 30 12:51:30.154120 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.154102 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-cabundle\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:30.156591 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.156563 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-key\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:30.177028 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.177005 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2m8\" (UniqueName: \"kubernetes.io/projected/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-kube-api-access-dm2m8\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:30.181998 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.181970 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 12:51:30.182518 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.182498 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/0.log" Oct 30 12:51:30.182615 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.182559 2569 generic.go:356] "Generic (PLEG): container finished" podID="264f3a6d-7988-4ee8-bf15-bf13e991a63c" containerID="cf5f0644426bbd326a55d3900f2557b6ff58232881822a9651d34dd39b748bec" exitCode=255 Oct 30 12:51:30.182727 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.182688 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" event={"ID":"264f3a6d-7988-4ee8-bf15-bf13e991a63c","Type":"ContainerDied","Data":"cf5f0644426bbd326a55d3900f2557b6ff58232881822a9651d34dd39b748bec"} Oct 30 12:51:30.182777 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.182754 2569 scope.go:122] "RemoveContainer" containerID="bd7b99f9ffe1380bb0b0c5216b5b2526ee2ed13cad791c7b19be44cd656bb4d9" Oct 30 12:51:30.183372 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.183114 2569 scope.go:122] "RemoveContainer" containerID="cf5f0644426bbd326a55d3900f2557b6ff58232881822a9651d34dd39b748bec" Oct 30 12:51:30.183463 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:30.183403 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l2xkr_openshift-console-operator(264f3a6d-7988-4ee8-bf15-bf13e991a63c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" podUID="264f3a6d-7988-4ee8-bf15-bf13e991a63c" Oct 30 12:51:30.184477 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.184454 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww" event={"ID":"da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac","Type":"ContainerStarted","Data":"f18c301930b64494b7e5fb40bd01304e0d7f4942803d56b63f06f74ced418f47"} Oct 30 12:51:30.185745 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.185725 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv" event={"ID":"8ebe4466-834b-4748-9fa5-6a01043ae3ad","Type":"ContainerStarted","Data":"9f96260c566a49fbd2b3050c8111f1c6fba63c44108c4e6d1ebb0c8391d3f12f"} Oct 30 12:51:30.187252 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.187228 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7mcfh-fnbft" event={"ID":"4f13020b-562d-470d-9bc4-70a5dd26f1e9","Type":"ContainerStarted","Data":"6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2"} Oct 30 12:51:30.188811 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.188780 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b9v2t" event={"ID":"b6151a5c-6392-4151-8c74-6d518f34b4d0","Type":"ContainerStarted","Data":"2a8cea131ad9f9839f13de2d16984886e4a0223c2c3a661760b4759eee26f2a4"} Oct 30 12:51:30.223298 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.223270 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:30.254667 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.254643 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:30.272374 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.272308 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7mcfh-fnbft" podStartSLOduration=2.272292669 podStartE2EDuration="2.272292669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:30.269766899 +0000 UTC m=+38.848456455" watchObservedRunningTime="2026-10-30 12:51:30.272292669 +0000 UTC m=+38.850982234" Oct 30 12:51:30.309003 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.308345 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b9v2t" podStartSLOduration=34.964763361 podStartE2EDuration="38.308309953s" totalImagesPullingTime="3.343546592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:30.307483391 +0000 UTC m=+38.886172952" watchObservedRunningTime="2026-10-30 12:51:30.308309953 +0000 UTC m=+38.886999518" Oct 30 12:51:30.422980 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.422893 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:30.554490 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:30.554331 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xzkxq"] Oct 30 12:51:31.198105 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:31.198072 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 12:51:31.198551 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:31.198489 2569 scope.go:122] "RemoveContainer" containerID="cf5f0644426bbd326a55d3900f2557b6ff58232881822a9651d34dd39b748bec" Oct 30 12:51:31.198678 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:31.198658 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l2xkr_openshift-console-operator(264f3a6d-7988-4ee8-bf15-bf13e991a63c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" podUID="264f3a6d-7988-4ee8-bf15-bf13e991a63c" Oct 30 12:51:31.202132 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:31.202107 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" event={"ID":"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9","Type":"ContainerStarted","Data":"550ca391f48edf94d1caa774cc5e1ba08ca6f9255f8884fc1fa3cb643b2f64d5"} Oct 30 12:51:31.202240 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:31.202151 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" event={"ID":"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9","Type":"ContainerStarted","Data":"6d188dbb1dfaa2918020a85c492e254dda979525e8512c9e6d640fc2b8cbcc91"} Oct 30 12:51:31.202479 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:31.202434 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:31.484944 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:31.484854 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:31.739609 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:31.739531 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:31.988423 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:31.988390 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:32.080170 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.080138 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:32.080306 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.080194 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:32.080306 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:32.080240 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:32.080306 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:32.080295 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:36.080281013 +0000 UTC m=+44.658970560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:32.080475 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:32.080311 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:51:32.080475 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:32.080377 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:36.080360748 +0000 UTC m=+44.659050311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : configmap "kube-rbac-proxy" not found Oct 30 12:51:32.094363 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.094312 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v4l8n_d0471989-f151-47a5-9d39-97d85760da9e/dns-node-resolver/0.log" Oct 30 12:51:32.147044 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.147017 2569 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:32.207446 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.207380 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv" event={"ID":"8ebe4466-834b-4748-9fa5-6a01043ae3ad","Type":"ContainerStarted","Data":"3031eb45e1d09110150085ea45dc1f32131af98c4c056569f31dca90b5b1b101"} Oct 30 12:51:32.210033 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.209976 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww" event={"ID":"da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac","Type":"ContainerStarted","Data":"0b2173811b782f5e913e44a4e9b6f1b89e9ec86d75b1748419b4e948506c6ce4"} Oct 30 12:51:32.210903 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.210875 2569 scope.go:122] "RemoveContainer" containerID="cf5f0644426bbd326a55d3900f2557b6ff58232881822a9651d34dd39b748bec" Oct 30 12:51:32.211068 ip-10-0-133-61 kubenswrapper[2569]: E1030 12:51:32.211048 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l2xkr_openshift-console-operator(264f3a6d-7988-4ee8-bf15-bf13e991a63c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" podUID="264f3a6d-7988-4ee8-bf15-bf13e991a63c" Oct 30 12:51:32.319171 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.317967 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" podStartSLOduration=3.317945044 podStartE2EDuration="3.317945044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:31.305696507 +0000 UTC m=+39.884386073" watchObservedRunningTime="2026-10-30 12:51:32.317945044 +0000 UTC m=+40.896634610" Oct 30 12:51:32.319171 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.318213 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv" podStartSLOduration=1.435121313 podStartE2EDuration="3.318203883s" totalImagesPullingTime="1.88308257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:32.317173413 +0000 UTC m=+40.895862978" watchObservedRunningTime="2026-10-30 12:51:32.318203883 +0000 UTC m=+40.896893449" Oct 30 12:51:32.481547 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.481507 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:32.690538 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.690475 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6p76v_126669c1-ee47-49ab-a6f3-3592a66b1fd2/node-ca/0.log" Oct 30 12:51:32.911445 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:32.911416 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:33.101182 ip-10-0-133-61 kubenswrapper[2569]: I1030 12:51:33.101118 2569 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:33.101273 ip-10-0-133-61 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 12:51:33.120717 ip-10-0-133-61 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 12:51:33.120889 ip-10-0-133-61 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 12:51:33.120941 ip-10-0-133-61 systemd[1]: kubelet.service: Consumed 3.290s CPU time. Oct 30 12:51:33.134915 ip-10-0-133-61 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:51:33.241704 ip-10-0-133-61 kubenswrapper[5049]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:33.241704 ip-10-0-133-61 kubenswrapper[5049]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:51:33.241704 ip-10-0-133-61 kubenswrapper[5049]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:33.241704 ip-10-0-133-61 kubenswrapper[5049]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:33.243828 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243741 5049 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:33.243828 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243827 5049 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:33.243828 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243831 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243835 5049 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243839 5049 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243842 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243845 5049 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243849 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243852 5049 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243855 5049 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243858 5049 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243861 5049 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243863 5049 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243866 5049 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243870 5049 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243875 5049 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243879 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243882 5049 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243885 5049 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243889 5049 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243892 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:33.243922 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243894 5049 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243898 5049 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243900 5049 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243903 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243906 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243909 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243916 5049 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243919 5049 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243922 5049 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243925 5049 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243928 5049 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243931 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243934 5049 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243937 5049 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243940 5049 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243942 5049 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243945 5049 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243947 5049 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243950 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:33.244424 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243953 5049 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243955 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243958 5049 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243961 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243963 5049 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243966 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243969 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243971 5049 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243974 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243977 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243979 5049 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243983 5049 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243986 5049 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243989 5049 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243991 5049 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243994 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.243997 5049 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244000 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244003 5049 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:33.244874 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244006 5049 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244009 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244012 5049 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244015 5049 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244018 5049 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244020 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244023 5049 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244027 5049 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244030 5049 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244033 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244036 5049 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244039 5049 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244042 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244045 5049 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244048 5049 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244050 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244053 5049 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244056 5049 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244058 5049 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244061 5049 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:33.245355 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244064 5049 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244067 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244070 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244073 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244075 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244079 5049 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244081 5049 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244084 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244087 5049 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244090 5049 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244093 5049 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244096 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244098 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244101 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244104 5049 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244107 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244110 5049 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244112 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244115 5049 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:33.245879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244118 5049 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244120 5049 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244123 5049 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244126 5049 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244128 5049 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244131 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244134 5049 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244138 5049 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244928 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244937 5049 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244940 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244944 5049 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244947 5049 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244950 5049 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244954 5049 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244959 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244962 5049 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244965 5049 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244968 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244971 5049 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:33.246344 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244975 5049 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244977 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244980 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244983 5049 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244986 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244988 5049 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244992 5049 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.244995 5049 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245003 5049 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245007 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245010 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245012 5049 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245015 5049 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245017 5049 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245020 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245023 5049 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245025 5049 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245028 5049 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245031 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:33.246880 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245034 5049 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245037 5049 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245040 5049 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245042 5049 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245046 5049 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245048 5049 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245051 5049 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245054 5049 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245056 5049 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245059 5049 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245062 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245064 5049 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245067 5049 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245070 5049 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245072 5049 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245075 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245078 5049 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245081 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245085 5049 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245090 5049 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:33.247345 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245093 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245096 5049 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245104 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245107 5049 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245110 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245113 5049 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245115 5049 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245118 5049 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245121 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245123 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245126 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245129 5049 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245132 5049 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245135 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245138 5049 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245141 5049 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245144 5049 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245147 5049 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245150 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245152 5049 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:33.247831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245155 5049 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245157 5049 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245160 5049 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245163 5049 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245165 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245168 5049 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245171 5049 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245173 5049 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245176 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245178 5049 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245181 5049 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245184 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245186 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245189 5049 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245191 5049 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245200 5049 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245203 5049 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245209 5049 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245211 5049 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245214 5049 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:33.248377 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245217 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245220 5049 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245222 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245225 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245228 5049 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245230 5049 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245233 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245236 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245238 5049 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245241 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245244 5049 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245246 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245249 5049 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245252 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245254 5049 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245785 5049 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245793 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245797 5049 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245800 5049 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:33.248908 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245802 5049 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245805 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245808 5049 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245811 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245814 5049 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245816 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245819 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245822 5049 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245826 5049 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245829 5049 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245834 5049 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245837 5049 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245839 5049 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245842 5049 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245845 5049 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245848 5049 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245851 5049 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245854 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245856 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245859 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:33.249388 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245862 5049 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245864 5049 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245867 5049 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245870 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245873 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245875 5049 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245878 5049 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245881 5049 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245884 5049 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245887 5049 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245890 5049 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245893 5049 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245896 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245898 5049 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245901 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245904 5049 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245906 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245909 5049 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245911 5049 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245914 5049 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:33.249871 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245917 5049 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245920 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245924 5049 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245927 5049 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245930 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245932 5049 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245935 5049 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245939 5049 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245943 5049 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245947 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245950 5049 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245953 5049 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245956 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245959 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245962 5049 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245964 5049 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245967 5049 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245970 5049 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245972 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:33.250631 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245975 5049 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245978 5049 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245980 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245983 5049 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245986 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245988 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245991 5049 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245994 5049 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.245997 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246000 5049 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246002 5049 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246005 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246008 5049 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246010 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246013 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246017 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246021 5049 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246023 5049 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246026 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:33.251353 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246030 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246033 5049 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246035 5049 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246038 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246042 5049 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246046 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246049 5049 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246052 5049 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246054 5049 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246057 5049 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246060 5049 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246063 5049 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246078 5049 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246081 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246085 5049 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246088 5049 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246090 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246093 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246096 5049 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246098 5049 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:33.251838 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246101 5049 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246104 5049 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246106 5049 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246109 5049 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246196 5049 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246203 5049 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246218 5049 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246222 5049 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246229 5049 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246233 5049 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246237 5049 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246241 5049 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246247 5049 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246250 5049 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246254 5049 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246257 5049 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246261 5049 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246264 5049 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246267 5049 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246271 5049 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246274 5049 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246277 5049 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246280 5049 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246283 5049 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:51:33.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246287 5049 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246290 5049 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246294 5049 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246298 5049 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246301 5049 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246304 5049 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246308 5049 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246311 5049 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246314 5049 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246331 5049 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246334 5049 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246337 5049 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246342 5049 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246346 5049 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246349 5049 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246353 5049 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246357 5049 flags.go:64] FLAG: --enable-server="true" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246360 5049 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246365 5049 flags.go:64] FLAG: --event-burst="100" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246368 5049 flags.go:64] FLAG: --event-qps="50" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246373 5049 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246376 5049 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246379 5049 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246383 5049 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:51:33.252887 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246386 5049 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246390 5049 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246393 5049 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246396 5049 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246399 5049 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246402 5049 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246405 5049 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246408 5049 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246411 5049 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246414 5049 flags.go:64] FLAG: --feature-gates="" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246418 5049 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246422 5049 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246425 5049 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246428 5049 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246432 5049 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246435 5049 flags.go:64] FLAG: --help="false" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246438 5049 flags.go:64] FLAG: --hostname-override="ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246441 5049 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246444 5049 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246448 5049 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246451 5049 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246454 5049 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246458 5049 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:51:33.253525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246461 5049 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246464 5049 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246467 5049 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246470 5049 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246474 5049 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246478 5049 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246482 5049 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246485 5049 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246488 5049 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246491 5049 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246494 5049 flags.go:64] FLAG: --lock-file="" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246497 5049 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246500 5049 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246503 5049 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246508 5049 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246511 5049 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246514 5049 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246517 5049 flags.go:64] FLAG: --logging-format="text" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246520 5049 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246524 5049 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246527 5049 flags.go:64] FLAG: --manifest-url="" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246530 5049 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246535 5049 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246538 5049 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246542 5049 flags.go:64] FLAG: --max-pods="110" Oct 30 12:51:33.254068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246545 5049 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246548 5049 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246551 5049 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246554 5049 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246557 5049 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246560 5049 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246563 5049 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246570 5049 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246573 5049 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246576 5049 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246579 5049 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246582 5049 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246588 5049 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246591 5049 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246594 5049 flags.go:64] FLAG: --port="10250" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246597 5049 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246600 5049 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c12c475088b1ba9a" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246603 5049 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246606 5049 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246609 5049 flags.go:64] FLAG: --register-node="true" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246612 5049 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246616 5049 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246619 5049 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246622 5049 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246625 5049 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246629 5049 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:51:33.254699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246632 5049 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246635 5049 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246638 5049 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246641 5049 flags.go:64] FLAG: --runonce="false" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246645 5049 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246648 5049 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246651 5049 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246654 5049 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246657 5049 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246660 5049 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246664 5049 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246667 5049 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246670 5049 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246673 5049 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246676 5049 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246679 5049 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246682 5049 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246685 5049 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246689 5049 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246696 5049 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246700 5049 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246703 5049 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246706 5049 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246709 5049 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246712 5049 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:51:33.255340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246715 5049 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246719 5049 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246722 5049 flags.go:64] FLAG: --v="2" Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246727 5049 flags.go:64] FLAG: --version="false" Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246731 5049 flags.go:64] FLAG: --vmodule="" Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246735 5049 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.246738 5049 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246880 5049 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246884 5049 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246888 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246892 5049 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246896 5049 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246900 5049 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246904 5049 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246907 5049 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246910 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246913 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246915 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246918 5049 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246921 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246923 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:33.255990 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246926 5049 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246929 5049 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246932 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246934 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246938 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246943 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246946 5049 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246949 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246951 5049 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246954 5049 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246956 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246959 5049 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246962 5049 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246964 5049 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246967 5049 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246970 5049 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246973 5049 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246975 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246978 5049 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246981 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:33.256505 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246983 5049 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246986 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246989 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246991 5049 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246994 5049 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.246997 5049 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247000 5049 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247002 5049 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247005 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247008 5049 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247011 5049 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247013 5049 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247016 5049 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247019 5049 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247021 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247024 5049 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247028 5049 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247032 5049 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247035 5049 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:33.257148 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247038 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247041 5049 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247044 5049 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247046 5049 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247049 5049 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247052 5049 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247054 5049 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247057 5049 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247060 5049 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247062 5049 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247065 5049 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247068 5049 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247070 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247073 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247075 5049 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247078 5049 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247081 5049 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247083 5049 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247086 5049 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247089 5049 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:33.257715 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247091 5049 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247094 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247096 5049 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247099 5049 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247101 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247104 5049 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247107 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247109 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247112 5049 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247116 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247121 5049 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247125 5049 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247128 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247134 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247137 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247140 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247143 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247146 5049 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247149 5049 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:33.258413 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247152 5049 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247154 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247157 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247160 5049 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247163 5049 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247165 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247168 5049 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247171 5049 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247173 5049 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247176 5049 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247178 5049 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247181 5049 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247184 5049 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247186 5049 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.247212 5049 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247247 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:33.258925 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247251 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247255 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247258 5049 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247261 5049 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247263 5049 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247266 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247270 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247273 5049 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247277 5049 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247280 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247283 5049 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247286 5049 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247289 5049 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247292 5049 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247294 5049 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247298 5049 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247300 5049 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247303 5049 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247306 5049 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247309 5049 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:33.259309 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247311 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247314 5049 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247330 5049 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247335 5049 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247338 5049 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247341 5049 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247344 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247347 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247350 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247353 5049 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247355 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247358 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247361 5049 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247364 5049 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247367 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247370 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247372 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247375 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247379 5049 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:33.259916 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247382 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247386 5049 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247389 5049 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247392 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247395 5049 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247398 5049 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247401 5049 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247404 5049 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247407 5049 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247410 5049 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247413 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247415 5049 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247418 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247420 5049 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247423 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247426 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247428 5049 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247431 5049 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247434 5049 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247437 5049 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:33.260521 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247440 5049 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247443 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247445 5049 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247448 5049 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247451 5049 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247453 5049 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247456 5049 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247459 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247461 5049 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247464 5049 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247467 5049 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247471 5049 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247474 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247478 5049 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247481 5049 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247485 5049 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247487 5049 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247490 5049 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247493 5049 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:33.261010 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247496 5049 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247499 5049 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247501 5049 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247504 5049 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247507 5049 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247510 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247512 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247515 5049 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247518 5049 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247520 5049 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247523 5049 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247526 5049 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247529 5049 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247531 5049 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247534 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247537 5049 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247539 5049 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247542 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247545 5049 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:33.261556 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247547 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247550 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247553 5049 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247555 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247560 5049 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247565 5049 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247577 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.247582 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.252616 5049 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.252635 5049 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.252652 5049 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.252658 5049 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252718 5049 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252723 5049 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252728 5049 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252733 5049 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252736 5049 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252739 5049 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252742 5049 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:33.262047 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252745 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252748 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252751 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252754 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252757 5049 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252760 5049 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252763 5049 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252766 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252769 5049 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252772 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252775 5049 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252778 5049 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252781 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252784 5049 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252787 5049 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252790 5049 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252793 5049 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252795 5049 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252799 5049 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252801 5049 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:33.262586 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252804 5049 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252807 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252810 5049 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252813 5049 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252816 5049 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252819 5049 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252822 5049 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252825 5049 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252828 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252830 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252833 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252836 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252839 5049 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252841 5049 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252844 5049 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252847 5049 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252850 5049 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252852 5049 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252855 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:33.263133 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252859 5049 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252862 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252865 5049 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252868 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252870 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252873 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252876 5049 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252879 5049 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252881 5049 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252884 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252888 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252891 5049 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252895 5049 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252899 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252902 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252905 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252908 5049 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252911 5049 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252914 5049 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:33.263662 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252916 5049 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252919 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252922 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252924 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252927 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252931 5049 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252934 5049 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252936 5049 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252939 5049 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252942 5049 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252945 5049 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252947 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252950 5049 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252953 5049 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252956 5049 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252960 5049 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252963 5049 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252965 5049 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252969 5049 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:33.264229 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252971 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252974 5049 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252977 5049 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252979 5049 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252982 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252985 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252988 5049 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252990 5049 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252993 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252996 5049 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.252999 5049 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253001 5049 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253004 5049 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253007 5049 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253010 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253012 5049 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253015 5049 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253018 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253021 5049 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253024 5049 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:33.264691 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253026 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253029 5049 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.253059 5049 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253203 5049 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253222 5049 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253225 5049 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253229 5049 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253232 5049 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253236 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253239 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253242 5049 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253245 5049 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253247 5049 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253250 5049 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253253 5049 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253256 5049 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:33.265160 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253258 5049 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253261 5049 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253264 5049 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253266 5049 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253269 5049 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253272 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253275 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253277 5049 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253280 5049 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253283 5049 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253286 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253289 5049 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253292 5049 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253295 5049 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253298 5049 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253302 5049 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253306 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253309 5049 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253312 5049 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:33.265899 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253315 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253332 5049 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253336 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253338 5049 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253341 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253344 5049 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253348 5049 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253351 5049 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253354 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253357 5049 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253359 5049 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253362 5049 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253365 5049 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253367 5049 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253370 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253373 5049 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253376 5049 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253378 5049 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253381 5049 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253384 5049 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:33.266535 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253387 5049 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253389 5049 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253392 5049 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253395 5049 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253398 5049 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253400 5049 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253403 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253406 5049 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253409 5049 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253412 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253414 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253417 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253422 5049 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253426 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253429 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253432 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253435 5049 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253437 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253440 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:33.267008 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253444 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253446 5049 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253449 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253452 5049 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253455 5049 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253458 5049 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253460 5049 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253463 5049 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253465 5049 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253468 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253471 5049 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253473 5049 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253476 5049 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253479 5049 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253481 5049 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253484 5049 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253487 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253492 5049 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253495 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253497 5049 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:33.267670 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253500 5049 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253503 5049 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253506 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253509 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253512 5049 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253515 5049 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253518 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253520 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253523 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253526 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253529 5049 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253532 5049 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253535 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253539 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253542 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.253566 5049 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:33.268147 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253599 5049 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253603 5049 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253606 5049 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253610 5049 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253613 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253615 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253620 5049 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253622 5049 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253625 5049 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253628 5049 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253630 5049 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253633 5049 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253636 5049 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253638 5049 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253642 5049 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253645 5049 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253647 5049 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253650 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253653 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253655 5049 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:33.268548 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253658 5049 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253661 5049 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253664 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253666 5049 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253669 5049 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253672 5049 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253675 5049 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253679 5049 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253683 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253686 5049 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253689 5049 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253692 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253695 5049 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253697 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253700 5049 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253702 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253705 5049 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253708 5049 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253712 5049 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253715 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:33.269061 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253717 5049 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253720 5049 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253723 5049 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253725 5049 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253728 5049 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253731 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253733 5049 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253736 5049 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253739 5049 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253741 5049 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253744 5049 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253747 5049 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253750 5049 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253752 5049 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253755 5049 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253757 5049 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253760 5049 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253763 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253765 5049 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:33.269658 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253768 5049 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253770 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253773 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253776 5049 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253781 5049 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253785 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253788 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253791 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253794 5049 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253797 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253799 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253804 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253806 5049 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253809 5049 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253812 5049 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253815 5049 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253817 5049 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253820 5049 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253823 5049 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:33.270295 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253826 5049 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253829 5049 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253831 5049 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253834 5049 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253836 5049 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253840 5049 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253842 5049 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253845 5049 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253847 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253850 5049 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253853 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253855 5049 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253858 5049 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253861 5049 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253863 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253866 5049 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253868 5049 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253871 5049 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253875 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253878 5049 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:33.270971 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253881 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253883 5049 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253886 5049 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253890 5049 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253894 5049 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253896 5049 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253899 5049 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:33.253901 5049 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.253968 5049 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.254989 5049 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.255047 5049 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.255235 5049 server.go:1014] "Starting client certificate rotation" Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.255342 5049 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.255384 5049 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:45:51 +0000 UTC" deadline="2028-07-07 00:42:33.083848082 +0000 UTC" Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.255418 5049 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14771h50m59.828431967s" Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.255704 5049 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:33.271669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.256748 5049 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:33.272105 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.258059 5049 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:51:33.272105 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.259756 5049 log.go:25] "Validated CRI v1 image API" Oct 30 12:51:33.272105 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.260478 5049 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:51:33.272105 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.262452 5049 fs.go:135] Filesystem UUIDs: map[5da986bb-eb87-4471-8b83-e126387a21ba:/dev/nvme0n1p4 65cbc65c-dc77-4a3f-bc0f-576f44b51184:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 12:51:33.272236 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.262472 5049 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/027a98a5d9752e6e47d1a3545787666832bc6f7d9dc2a465428d4c3e538577f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/027a98a5d9752e6e47d1a3545787666832bc6f7d9dc2a465428d4c3e538577f9/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/129bffca37c4391ace11af0211c2545e03d673fd42f09996673efc220baa21d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/129bffca37c4391ace11af0211c2545e03d673fd42f09996673efc220baa21d8/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12f9bb1918bc92524b935d14cd072890cddf9faf0646ad13663cbe23fcf18d10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12f9bb1918bc92524b935d14cd072890cddf9faf0646ad13663cbe23fcf18d10/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/250d94911962dbd46d73c952d9be812d6870122389f57106b8d8a3d4bad7eac6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/250d94911962dbd46d73c952d9be812d6870122389f57106b8d8a3d4bad7eac6/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3956a38b2219f01a26c99d1ea7cb75dfeaefd9601fc67f4398a1465d239d0062/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3956a38b2219f01a26c99d1ea7cb75dfeaefd9601fc67f4398a1465d239d0062/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ca4804dd5facaee14638f9bffe0814faa9af1cd905a1d446a36a59b183a1920/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ca4804dd5facaee14638f9bffe0814faa9af1cd905a1d446a36a59b183a1920/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/565530e1715a000908604d622d667bbca327e49f621cf4c1c0cd1bc52f64b7e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/565530e1715a000908604d622d667bbca327e49f621cf4c1c0cd1bc52f64b7e0/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60a8edae6ab58dbde1749a7fe3300f27610a9f73b828c178f347fe8191447be0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60a8edae6ab58dbde1749a7fe3300f27610a9f73b828c178f347fe8191447be0/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60edcebe872bf55f027cc442100417d57793ec9b989f58f1b541cdb7ecec3b2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60edcebe872bf55f027cc442100417d57793ec9b989f58f1b541cdb7ecec3b2e/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64a2cfedc1955d48656db6ae6abe61e0e481cc1c4f72783e9e5a30f386901db9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64a2cfedc1955d48656db6ae6abe61e0e481cc1c4f72783e9e5a30f386901db9/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65256a7c5313aee03415a40f1bae0a0424acc69602780bc70d87be25fdb5911c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65256a7c5313aee03415a40f1bae0a0424acc69602780bc70d87be25fdb5911c/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d188dbb1dfaa2918020a85c492e254dda979525e8512c9e6d640fc2b8cbcc91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d188dbb1dfaa2918020a85c492e254dda979525e8512c9e6d640fc2b8cbcc91/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76b60094adf3e50acd5ec3de283723592c4ad82d118f03a96a6c93ef5152f110/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76b60094adf3e50acd5ec3de283723592c4ad82d118f03a96a6c93ef5152f110/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d67e7e634aaf4db20f4c3d88b3b869f823864006342c66cd3918339f37396b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d67e7e634aaf4db20f4c3d88b3b869f823864006342c66cd3918339f37396b7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f96260c566a49fbd2b3050c8111f1c6fba63c44108c4e6d1ebb0c8391d3f12f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f96260c566a49fbd2b3050c8111f1c6fba63c44108c4e6d1ebb0c8391d3f12f/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b665cb8d66f7fa16481c292520994f406684d6ae84ce19bf7271a5396956cf25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b665cb8d66f7fa16481c292520994f406684d6ae84ce19bf7271a5396956cf25/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc486ad4df2adafbc04482c3cc486169273ab8751d5532c121849215ff6bef61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc486ad4df2adafbc04482c3cc486169273ab8751d5532c121849215ff6bef61/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d24bd95c7d6564ae3fe6506b5002a9f05323f62aa5abd3f4526c8f0ddf5a45ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d24bd95c7d6564ae3fe6506b5002a9f05323f62aa5abd3f4526c8f0ddf5a45ae/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d29a281f0d2d0407f08ee5cda8a9ab237a8fa4ba655bc351fa5033dc58e47186/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d29a281f0d2d0407f08ee5cda8a9ab237a8fa4ba655bc351fa5033dc58e47186/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da0500037ffeac56eb34a48abc809696866cfc8cf4463187dcc189cce7bd4659/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da0500037ffeac56eb34a48abc809696866cfc8cf4463187dcc189cce7bd4659/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0c544b2eaa4ce3ca8a3893f688d1004913ced0c0c32744fd6874c8a4f520dc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0c544b2eaa4ce3ca8a3893f688d1004913ced0c0c32744fd6874c8a4f520dc0/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef12761b54de310bb0455cd4b2c0354e74c304518a7bdb116398633ff40056ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef12761b54de310bb0455cd4b2c0354e74c304518a7bdb116398633ff40056ad/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efe28e4e8d6e998580d6116017ab7ce7ff749ca4595eb34f4089e8e6b4aa2ec7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efe28e4e8d6e998580d6116017ab7ce7ff749ca4595eb34f4089e8e6b4aa2ec7/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f18c301930b64494b7e5fb40bd01304e0d7f4942803d56b63f06f74ced418f47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f18c301930b64494b7e5fb40bd01304e0d7f4942803d56b63f06f74ced418f47/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f40308f148529bb5b59cc5f21ec25642c8f89bc01f8f39ecddf49e7f2dfcc3c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f40308f148529bb5b59cc5f21ec25642c8f89bc01f8f39ecddf49e7f2dfcc3c6/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcc75f970f6ed346b1d2485c1135a8ff4551b9c0e2b9cbc7af01538a61b9779f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcc75f970f6ed346b1d2485c1135a8ff4551b9c0e2b9cbc7af01538a61b9779f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3/volumes/kubernetes.io~projected/kube-api-access-h7t4r:{mountpoint:/var/lib/kubelet/pods/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3/volumes/kubernetes.io~projected/kube-api-access-h7t4r major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cbb909f-28ac-4b1e-9a95-5b8870886b61/volumes/kubernetes.io~projected/kube-api-access-mpqh6:{mountpoint:/var/lib/kubelet/pods/0cbb909f-28ac-4b1e-9a95-5b8870886b61/volumes/kubernetes.io~projected/kube-api-access-mpqh6 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cbb909f-28ac-4b1e-9a95-5b8870886b61/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0cbb909f-28ac-4b1e-9a95-5b8870886b61/volumes/kubernetes.io~secret/cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/116675da-08c0-4101-835c-48193661ccbf/volumes/kubernetes.io~projected/kube-api-access-dcdfs:{mountpoint:/var/lib/kubelet/pods/116675da-08c0-4101-835c-48193661ccbf/volumes/kubernetes.io~projected/kube-api-access-dcdfs major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/126669c1-ee47-49ab-a6f3-3592a66b1fd2/volumes/kubernetes.io~projected/kube-api-access-dcd62:{mountpoint:/var/lib/kubelet/pods/126669c1-ee47-49ab-a6f3-3592a66b1fd2/volumes/kubernetes.io~projected/kube-api-access-dcd62 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15a59c67-94e0-48e5-b8ff-28a55dea0996/volumes/kubernetes.io~projected/kube-api-access-r774z:{mountpoint:/var/lib/kubelet/pods/15a59c67-94e0-48e5-b8ff-28a55dea0996/volumes/kubernetes.io~projected/kube-api-access-r774z major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/173ce15b-35dd-4a98-8f29-d13c62b56206/volumes/kubernetes.io~projected/kube-api-access-sq4nn:{mountpoint:/var/lib/kubelet/pods/173ce15b-35dd-4a98-8f29-d13c62b56206/volumes/kubernetes.io~projected/kube-api-access-sq4nn major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/173ce15b-35dd-4a98-8f29-d13c62b56206/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/173ce15b-35dd-4a98-8f29-d13c62b56206/volumes/kubernetes.io~secret/default-certificate major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/173ce15b-35dd-4a98-8f29-d13c62b56206/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/173ce15b-35dd-4a98-8f29-d13c62b56206/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~projected/kube-api-access-s6ml9:{mountpoint:/var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~projected/kube-api-access-s6ml9 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f/volumes/kubernetes.io~projected/kube-api-access-ccvtd:{mountpoint:/var/lib/kubelet/pods/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f/volumes/kubernetes.io~projected/kube-api-access-ccvtd major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/264f3a6d-7988-4ee8-bf15-bf13e991a63c/volumes/kubernetes.io~projected/kube-api-access-j8xfl:{mountpoint:/var/lib/kubelet/pods/264f3a6d-7988-4ee8-bf15-bf13e991a63c/volumes/kubernetes.io~projected/kube-api-access-j8xfl major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/264f3a6d-7988-4ee8-bf15-bf13e991a63c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/264f3a6d-7988-4ee8-bf15-bf13e991a63c/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28a37fed-2af0-45d1-8e84-c658dbff6b61/volumes/kubernetes.io~projected/kube-api-access-zlxdg:{mountpoint:/var/lib/kubelet/pods/28a37fed-2af0-45d1-8e84-c658dbff6b61/volumes/kubernetes.io~projected/kube-api-access-zlxdg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~projected/kube-api-access-jqflv:{mountpoint:/var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~projected/kube-api-access-jqflv major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3400463b-e945-48d9-8dc5-8b8106cb6e26/volumes/kubernetes.io~projected/kube-api-access-8lvsp:{mountpoint:/var/lib/kubelet/pods/3400463b-e945-48d9-8dc5-8b8106cb6e26/volumes/kubernetes.io~projected/kube-api-access-8lvsp major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35925210-93d9-4a34-b733-6481da8921f3/volumes/kubernetes.io~projected/kube-api-access-npxxk:{mountpoint:/var/lib/kubelet/pods/35925210-93d9-4a34-b733-6481da8921f3/volumes/kubernetes.io~projected/kube-api-access-npxxk major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/482cc1b2-387f-4b1e-98f5-adada095ffec/volumes/kubernetes.io~projected/kube-api-access-c7m2h:{mountpoint:/var/lib/kubelet/pods/482cc1b2-387f-4b1e-98f5-adada095ffec/volumes/kubernetes.io~projected/kube-api-access-c7m2h major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f13020b-562d-470d-9bc4-70a5dd26f1e9/volumes/kubernetes.io~projected/kube-api-access-4k2f6:{mountpoint:/var/lib/kubelet/pods/4f13020b-562d-470d-9bc4-70a5dd26f1e9/volumes/kubernetes.io~projected/kube-api-access-4k2f6 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56a8078f-bd8e-42d8-86ef-4e0030e0ea76/volumes/kubernetes.io~projected/kube-api-access-ndg29:{mountpoint:/var/lib/kubelet/pods/56a8078f-bd8e-42d8-86ef-4e0030e0ea76/volumes/kubernetes.io~projected/kube-api-access-ndg29 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56a8078f-bd8e-42d8-86ef-4e0030e0ea76/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/56a8078f-bd8e-42d8-86ef-4e0030e0ea76/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58b3cad0-f9c1-4547-bb19-f1fd20041eca/volumes/kubernetes.io~projected/kube-api-access-qv69b:{mountpoint:/var/lib/kubelet/pods/58b3cad0-f9c1-4547-bb19-f1fd20041eca/volumes/kubernetes.io~projected/kube-api-access-qv69b major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe/volumes/kubernetes.io~projected/kube-api-access-ck6ww:{mountpoint:/var/lib/kubelet/pods/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe/volumes/kubernetes.io~projected/kube-api-access-ck6ww major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e6304cb-862e-43ea-8688-8298f61b8c5b/volumes/kubernetes.io~projected/kube-api-access-x7x4b:{mountpoint:/var/lib/kubelet/pods/6e6304cb-862e-43ea-8688-8298f61b8c5b/volumes/kubernetes.io~projected/kube-api-access-x7x4b major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ebe4466-834b-4748-9fa5-6a01043ae3ad/volumes/kubernetes.io~projected/kube-api-access-kjfn2:{mountpoint:/var/lib/kubelet/pods/8ebe4466-834b-4748-9fa5-6a01043ae3ad/volumes/kubernetes.io~projected/kube-api-access-kjfn2 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dc9e5e5-426c-4e0b-897f-54e0e28d3313/volumes/kubernetes.io~projected/kube-api-access-vkp4n:{mountpoint:/var/lib/kubelet/pods/9dc9e5e5-426c-4e0b-897f-54e0e28d3313/volumes/kubernetes.io~projected/kube-api-access-vkp4n major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0aa96f3-47c7-48f1-98a3-dd3c4548d593/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b0aa96f3-47c7-48f1-98a3-dd3c4548d593/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2767673-ded2-4026-8b3f-ff2582c0aede/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b2767673-ded2-4026-8b3f-ff2582c0aede/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2767673-ded2-4026-8b3f-ff2582c0aede/volumes/kubernetes.io~projected/kube-api-access-nx6xd:{mountpoint:/var/lib/kubelet/pods/b2767673-ded2-4026-8b3f-ff2582c0aede/volumes/kubernetes.io~projected/kube-api-access-nx6xd major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2767673-ded2-4026-8b3f-ff2582c0aede/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b2767673-ded2-4026-8b3f-ff2582c0aede/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3/volumes/kubernetes.io~projected/kube-api-access-qlchw:{mountpoint:/var/lib/kubelet/pods/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3/volumes/kubernetes.io~projected/kube-api-access-qlchw major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6151a5c-6392-4151-8c74-6d518f34b4d0/volumes/kubernetes.io~projected/kube-api-access-pm47z:{mountpoint:/var/lib/kubelet/pods/b6151a5c-6392-4151-8c74-6d518f34b4d0/volumes/kubernetes.io~projected/kube-api-access-pm47z major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbb88cf5-e088-4260-82e6-ed56a2b00176/volumes/kubernetes.io~projected/kube-api-access-pc2w4:{mountpoint:/var/lib/kubelet/pods/cbb88cf5-e088-4260-82e6-ed56a2b00176/volumes/kubernetes.io~projected/kube-api-access-pc2w4 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0471989-f151-47a5-9d39-97d85760da9e/volumes/kubernetes.io~projected/kube-api-access-s86zc:{mountpoint:/var/lib/kubelet/pods/d0471989-f151-47a5-9d39-97d85760da9e/volumes/kubernetes.io~projected/kube-api-access-s86zc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac/volumes/kubernetes.io~projected/kube-api-access-tvcdx:{mountpoint:/var/lib/kubelet/pods/da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac/volumes/kubernetes.io~projected/kube-api-access-tvcdx major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9/volumes/kubernetes.io~projected/kube-api-access-dm2m8:{mountpoint:/var/lib/kubelet/pods/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9/volumes/kubernetes.io~projected/kube-api-access-dm2m8 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0297b8f-b8e8-470e-a0cb-838697059ff4/volumes/kubernetes.io~projected/kube-api-access-5f27s:{mountpoint:/var/lib/kubelet/pods/f0297b8f-b8e8-470e-a0cb-838697059ff4/volumes/kubernetes.io~projected/kube-api-access-5f27s major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc1fa5e9-0704-4469-83b4-bdb65f799855/volumes/kubernetes.io~projected/kube-api-access-gsmn4:{mountpoint:/var/lib/kubelet/pods/fc1fa5e9-0704-4469-83b4-bdb65f799855/volumes/kubernetes.io~projected/kube-api-access-gsmn4 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc1fa5e9-0704-4469-83b4-bdb65f799855/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fc1fa5e9-0704-4469-83b4-bdb65f799855/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4/volumes/kubernetes.io~projected/kube-api-access-g99qq:{mountpoint:/var/lib/kubelet/pods/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4/volumes/kubernetes.io~projected/kube-api-access-g99qq major:0 minor:223 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5eca5f6c55f0c371f32614effb2f751a0ee2f2b14b79b621fd703d1417d29215/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e70ed8462ce8ef309ef68ada984f5cb8109f8c89c20020880bbe73160618b12b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/96388bfdb2547006fe425e45df184cf23fc4bb9ea25cffd70e0eff87737679cc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d704031ce4a65ff4639e6c011d7cd9b3d117f391b61cc8f0676dcfcfe965d210/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8a12e3adbb4d44fa0eaed4589aa19bca9259169cda5de62db8858af81e76c2ef/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fb96858bbb927363462b4a786215440783deba4b6d30fee49f7fd3366ee971bc/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8620140d5ee5c00e178eacaa309ef1b76ce2048127a93ac8dba6d3641155fd8e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3cd84555856267c65a9f22818fe2244aa21fdf587f7a143fef5897f0ad2f9397/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ec5d81973558baae96ba419f1ed33239f0476b19dbfac6d64cdabdd095b7a5b9/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/28e4fa5e9d34ce70022477764044efdb7c2f3fa5704113c16efb237d0b0eb64d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/0b29b6d4c19e9ffe0a94f4d9b5a9b142dfe21cb6b522a44dec610fb0ff491e89/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/1d60486933040ce2466781873b13ea037376587499b30aed17df0c80dad673d6/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ed9566226efcd8c6c732f8753a795d5545e7759ca378361bc4a2d3198da0e23f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ae0d970e7b9af167f3aef936eb79c04781b69f80bac7997415526ce9e56ac38c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/403bd9476438c532696934aceac256075a9228c82a958a6f7c1aa885f9eb2192/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cb8065dbc486cb0c22f2d9ef49e999b0b59a86345d7e0ffdbb4069a0f786c2dc/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/811bb08b9e4b1c7752c6c28e1e88bf9c924a4286b37af4711d0168c60439f998/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/44e540ed33e714009b13d83e59fb23842664d0217461c376ca22382679d8fd00/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c0943df44f840b0bab8db096032af5acfc1de3e9ec01be572981f93a4a05bf27/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b7ffce353e4f3be6632a397a897ee4ba4ca0bda5d20a2ec3025231356edd6c59/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1ec0ffe92dae228b205d1bfd89142c16f1e51411025de90d3be4093cb4b6cfaf/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/070635ac35a9f6d415152e5d0091512091bc434277b4596392004b759e8fccc3/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b4b408e218a59a1407557d230e8d5a896c58493bc7f452c473f80515496353d3/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/5d795761d4b9cf069154f01700a2199274b6938c8d3d4a6e1c6489fe402234cd/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/46beed8014b7ed1b7a27f3d15709a6a75e95449d61ccb01a8c2ab8b2204f16cc/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9354d3c94465c878ff9126047df480142ac42781c9cf14ff35096af74c987e85/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/15ad1cf4becaae28ea7c323f2bb16b0080952f16c69639b8123e2d17fbd17754/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/f340acc314970cc8d8253586f26eebefaed3e856991e5546e33c9f334550e0a3/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/8a941373ecebe9ce1322aa5e48a1ebebc62ca44cedc5efbee62c1698b29ab124/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/9f13b9cfcac794e372fa7bee6cb55da605b188f095d70929247cf17111bb71ef/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/d82991d3a815087a1b5629bc97c129b34514f480ff55840aa792373e2b42830b/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/adec5936ab156e214068c95fa652328a6b62a668cecb8e7fb67edc723a6b4f5b/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/5a5971fef3fb0febd8183962485bd489cdd21d0c981f04c44b7d71bf603924f4/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/991a7a71506ba0743f436de29aae7322f16f2fb529ee92d38913bd15ad688cc1/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/66cd83223747dd13d1ebc43610357ea3160fa7493faa880468570b163c9b8dfc/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/2b43066ae6d74c91d3d517feb731fa5a93b1b0ee47d4e642234a3aa030b28c17/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/28f757f16f3c7979a54a722196c65a3dc8183f555d53b8924026b3a86f31041c/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/2010168a57efa05b483881582b8ecd725e0b94cd1f2fae7e1097ddc40a90dab7/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d458cc3298fe7b62f1c401b8024ab877798d054a71ff8ee0378f2a7568a17750/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/00a18977bdfc98fad25e78a66a12fa48fa7cb4587dd4af59cabcf1bf4479cc37/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/25110693a36538475c2fe974194eac8e7043d06f9335beddb304769d1986a253/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/e8588a2e51a478c7f7cc5d3d6dcbae14d3e4fad78552bdc6933e0a7c98633bda/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/2e2872f26226051523eabd1bb54afeb166ea2e8901ff732bd5061d1651fb7282/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/6037d2d1268bdbf4e21f5e53f08047ff9f7c0f737757ceda1d84ccf5aca219ed/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/62218fbdff2c1294ccee1053c4ecdf7fcec255611b103ed538609ca6a6c6a0d8/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/3d6f7c85a7a0a10a9828027c3bd41bb0d5f0a04ae8aa860274e608b98c6d4d1a/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/d64afc35fe5a95f234b12a1f2a4d49faae6c65d4e4fd706c94ed2421ecc00869/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/4b97d3779971d5ae94e570b96dccd2645c32e67ab8b4821e65314746beab67db/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/fd06279b7f82a3635ff108255d57efffd411cf192a0ef96252bc2707dfe754e4/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/a565845006f0e41e653429c8c1b4210fa3f0b9f91329e571006d103745a1ad6e/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/b6a4b5e194b77af075c6b32899b0a6a2ebbb988be738704bee1ae5fcc4a6b9f8/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/26669ca2cbb0a0965b456c99c0891cb6c9fb5813fb57bccc13de7f1e5f0e8bae/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/aaa10eec68c89a9959e00205737ddc3f2705a919ea1c073a635c213beeb5323b/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/81ca5749f6091054a712a4885ae766c9346b3ccff96272a82869ec7769ac0dc1/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/fda7115ee594fa4821e2cc879b48796b33f3e9ae499865b285102e5571bfd947/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a34455c790b2a016b2f541cc5b8103d6ec38408e421fae428e3bc950c04f43e8/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/df642a954c12f254e49ddd9b0310ce821ff7965d8773505e549e29e5fd2556d9/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9339103af09489c489da7f11a5136583da53490e85b9264f8aa29f99a7546a06/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3c4688f152c8a603a375dc626804bdf431a8d8deeb5c49918abd413475ac75ad/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/03b551e20b5702cf3ddf4ceac3c50cf41ce9af9ad7b890d35f4b531c2580b8f5/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4cef1a0adcdc942cc1ccd0dd5735d040f4d1b2cbee4c6bc9eff291ac71c1b2bb/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bbc8b093d150af991e9e01e8c781379475fe96c5d25278e6756e213e113e33c6/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 12:51:33.272618 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.272212 5049 manager.go:217] Machine: {Timestamp:2026-10-30 12:51:33.271681457 +0000 UTC m=+0.086875871 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3165421 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23efe48c8cced7db791e27285b59cc SystemUUID:ec23efe4-8c8c-ced7-db79-1e27285b59cc BootID:1acb60c7-9a59-434b-89c8-2519d02f9c6c Filesystems:[{Device:/run/containers/storage/overlay-containers/64a2cfedc1955d48656db6ae6abe61e0e481cc1c4f72783e9e5a30f386901db9/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3/volumes/kubernetes.io~projected/kube-api-access-qlchw DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3956a38b2219f01a26c99d1ea7cb75dfeaefd9601fc67f4398a1465d239d0062/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cbb909f-28ac-4b1e-9a95-5b8870886b61/volumes/kubernetes.io~projected/kube-api-access-mpqh6 DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0297b8f-b8e8-470e-a0cb-838697059ff4/volumes/kubernetes.io~projected/kube-api-access-5f27s DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f/volumes/kubernetes.io~projected/kube-api-access-ccvtd DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/264f3a6d-7988-4ee8-bf15-bf13e991a63c/volumes/kubernetes.io~projected/kube-api-access-j8xfl DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbb88cf5-e088-4260-82e6-ed56a2b00176/volumes/kubernetes.io~projected/kube-api-access-pc2w4 DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2767673-ded2-4026-8b3f-ff2582c0aede/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15a59c67-94e0-48e5-b8ff-28a55dea0996/volumes/kubernetes.io~projected/kube-api-access-r774z DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/264f3a6d-7988-4ee8-bf15-bf13e991a63c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f13020b-562d-470d-9bc4-70a5dd26f1e9/volumes/kubernetes.io~projected/kube-api-access-4k2f6 DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef12761b54de310bb0455cd4b2c0354e74c304518a7bdb116398633ff40056ad/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b2767673-ded2-4026-8b3f-ff2582c0aede/volumes/kubernetes.io~projected/kube-api-access-nx6xd DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/027a98a5d9752e6e47d1a3545787666832bc6f7d9dc2a465428d4c3e538577f9/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/129bffca37c4391ace11af0211c2545e03d673fd42f09996673efc220baa21d8/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12f9bb1918bc92524b935d14cd072890cddf9faf0646ad13663cbe23fcf18d10/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/116675da-08c0-4101-835c-48193661ccbf/volumes/kubernetes.io~projected/kube-api-access-dcdfs DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc486ad4df2adafbc04482c3cc486169273ab8751d5532c121849215ff6bef61/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b2767673-ded2-4026-8b3f-ff2582c0aede/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b665cb8d66f7fa16481c292520994f406684d6ae84ce19bf7271a5396956cf25/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9/volumes/kubernetes.io~projected/kube-api-access-dm2m8 DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~projected/kube-api-access-s6ml9 DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ebe4466-834b-4748-9fa5-6a01043ae3ad/volumes/kubernetes.io~projected/kube-api-access-kjfn2 DeviceMajor:0 DeviceMinor:342 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~projected/kube-api-access-jqflv DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d67e7e634aaf4db20f4c3d88b3b869f823864006342c66cd3918339f37396b7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76b60094adf3e50acd5ec3de283723592c4ad82d118f03a96a6c93ef5152f110/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58b3cad0-f9c1-4547-bb19-f1fd20041eca/volumes/kubernetes.io~projected/kube-api-access-qv69b DeviceMajor:0 DeviceMinor:323 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0471989-f151-47a5-9d39-97d85760da9e/volumes/kubernetes.io~projected/kube-api-access-s86zc DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60edcebe872bf55f027cc442100417d57793ec9b989f58f1b541cdb7ecec3b2e/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35925210-93d9-4a34-b733-6481da8921f3/volumes/kubernetes.io~projected/kube-api-access-npxxk DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e6304cb-862e-43ea-8688-8298f61b8c5b/volumes/kubernetes.io~projected/kube-api-access-x7x4b DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4/volumes/kubernetes.io~projected/kube-api-access-g99qq DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56a8078f-bd8e-42d8-86ef-4e0030e0ea76/volumes/kubernetes.io~projected/kube-api-access-ndg29 DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65256a7c5313aee03415a40f1bae0a0424acc69602780bc70d87be25fdb5911c/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f40308f148529bb5b59cc5f21ec25642c8f89bc01f8f39ecddf49e7f2dfcc3c6/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b0aa96f3-47c7-48f1-98a3-dd3c4548d593/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/173ce15b-35dd-4a98-8f29-d13c62b56206/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0c544b2eaa4ce3ca8a3893f688d1004913ced0c0c32744fd6874c8a4f520dc0/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/28a37fed-2af0-45d1-8e84-c658dbff6b61/volumes/kubernetes.io~projected/kube-api-access-zlxdg DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56a8078f-bd8e-42d8-86ef-4e0030e0ea76/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/126669c1-ee47-49ab-a6f3-3592a66b1fd2/volumes/kubernetes.io~projected/kube-api-access-dcd62 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/173ce15b-35dd-4a98-8f29-d13c62b56206/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b6151a5c-6392-4151-8c74-6d518f34b4d0/volumes/kubernetes.io~projected/kube-api-access-pm47z DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc1fa5e9-0704-4469-83b4-bdb65f799855/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcc75f970f6ed346b1d2485c1135a8ff4551b9c0e2b9cbc7af01538a61b9779f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f96260c566a49fbd2b3050c8111f1c6fba63c44108c4e6d1ebb0c8391d3f12f/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/565530e1715a000908604d622d667bbca327e49f621cf4c1c0cd1bc52f64b7e0/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3/volumes/kubernetes.io~projected/kube-api-access-h7t4r DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/173ce15b-35dd-4a98-8f29-d13c62b56206/volumes/kubernetes.io~projected/kube-api-access-sq4nn DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3400463b-e945-48d9-8dc5-8b8106cb6e26/volumes/kubernetes.io~projected/kube-api-access-8lvsp DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d29a281f0d2d0407f08ee5cda8a9ab237a8fa4ba655bc351fa5033dc58e47186/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe/volumes/kubernetes.io~projected/kube-api-access-ck6ww DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc1fa5e9-0704-4469-83b4-bdb65f799855/volumes/kubernetes.io~projected/kube-api-access-gsmn4 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/482cc1b2-387f-4b1e-98f5-adada095ffec/volumes/kubernetes.io~projected/kube-api-access-c7m2h DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efe28e4e8d6e998580d6116017ab7ce7ff749ca4595eb34f4089e8e6b4aa2ec7/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da0500037ffeac56eb34a48abc809696866cfc8cf4463187dcc189cce7bd4659/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f18c301930b64494b7e5fb40bd01304e0d7f4942803d56b63f06f74ced418f47/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cbb909f-28ac-4b1e-9a95-5b8870886b61/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ca4804dd5facaee14638f9bffe0814faa9af1cd905a1d446a36a59b183a1920/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60a8edae6ab58dbde1749a7fe3300f27610a9f73b828c178f347fe8191447be0/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/250d94911962dbd46d73c952d9be812d6870122389f57106b8d8a3d4bad7eac6/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d24bd95c7d6564ae3fe6506b5002a9f05323f62aa5abd3f4526c8f0ddf5a45ae/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d188dbb1dfaa2918020a85c492e254dda979525e8512c9e6d640fc2b8cbcc91/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dc9e5e5-426c-4e0b-897f-54e0e28d3313/volumes/kubernetes.io~projected/kube-api-access-vkp4n DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac/volumes/kubernetes.io~projected/kube-api-access-tvcdx DeviceMajor:0 DeviceMinor:351 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:129bffca37c4391 MacAddress:2a:0b:f9:f3:ba:20 Speed:10000 Mtu:8901} {Name:12f9bb1918bc925 MacAddress:ea:59:98:e1:3c:cc Speed:10000 Mtu:8901} {Name:250d94911962dbd MacAddress:86:57:f8:3c:0f:47 Speed:10000 Mtu:8901} {Name:4ca4804dd5facae MacAddress:46:5d:af:51:24:e1 Speed:10000 Mtu:8901} {Name:65256a7c5313aee MacAddress:d6:24:66:fa:cd:8e Speed:10000 Mtu:8901} {Name:6d188dbb1dfaa29 MacAddress:ba:36:48:89:6a:24 Speed:10000 Mtu:8901} {Name:76b60094adf3e50 MacAddress:02:ba:b3:28:75:b8 Speed:10000 Mtu:8901} {Name:9f96260c566a49f MacAddress:22:a7:52:2f:9c:0e Speed:10000 Mtu:8901} {Name:b665cb8d66f7fa1 MacAddress:e6:64:f0:46:5e:94 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cb:1b:20:c5 Speed:0 Mtu:9001} {Name:br-int MacAddress:fe:d8:80:8c:11:35 Speed:0 Mtu:8901} {Name:d24bd95c7d6564a MacAddress:f2:91:b9:d2:72:c1 Speed:10000 Mtu:8901} {Name:da0500037ffeac5 MacAddress:e2:18:60:66:7a:c5 Speed:10000 Mtu:8901} {Name:e0c544b2eaa4ce3 MacAddress:8e:bf:5d:30:9f:73 Speed:10000 Mtu:8901} {Name:ef12761b54de310 MacAddress:be:3e:27:01:7c:f7 Speed:10000 Mtu:8901} {Name:efe28e4e8d6e998 MacAddress:ba:73:b0:0f:ab:f9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cb:1b:20:c5 Speed:0 Mtu:9001} {Name:f18c301930b6449 MacAddress:72:41:b9:a9:ad:85 Speed:10000 Mtu:8901} {Name:fcc75f970f6ed34 MacAddress:f6:a3:14:f9:61:dc Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:6a:94:2a:43:4a:71 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:9c:6b:4a:9e:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:51:33.272801 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.272623 5049 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:51:33.272801 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.272671 5049 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:51:33.273027 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273003 5049 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:51:33.273197 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273029 5049 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-61.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:51:33.273240 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273201 5049 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:51:33.273240 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273210 5049 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:51:33.273240 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273220 5049 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:51:33.273240 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273234 5049 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:51:33.273380 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273254 5049 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:51:33.273433 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273423 5049 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:51:33.273548 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273539 5049 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:51:33.273585 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273576 5049 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:51:33.273613 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273591 5049 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:51:33.273613 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273601 5049 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:51:33.273613 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273609 5049 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:51:33.273685 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.273617 5049 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:51:33.274592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.274575 5049 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:51:33.274592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.274592 5049 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:51:33.275138 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.275118 5049 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:51:33.278155 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278141 5049 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:51:33.278155 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278155 5049 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:51:33.278259 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278250 5049 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 12:51:33.278463 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278452 5049 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:51:33.278500 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278482 5049 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:51:33.278582 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278573 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:51:33.278611 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278588 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:51:33.278611 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278594 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:51:33.278611 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278607 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:51:33.278688 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278613 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:51:33.278688 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278622 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:51:33.278688 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278629 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:51:33.278688 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278635 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:51:33.278688 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278643 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:51:33.278688 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278648 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:51:33.278688 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278661 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:51:33.278688 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278670 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:51:33.278932 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278694 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:51:33.278932 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.278703 5049 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:51:33.282133 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.282120 5049 server.go:1263] "Started kubelet" Oct 30 12:51:33.282212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.282187 5049 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:51:33.282293 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.282241 5049 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:51:33.282673 ip-10-0-133-61 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:51:33.283386 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.283363 5049 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:51:33.284003 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.283659 5049 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:51:33.285236 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.285219 5049 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:51:33.288181 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.288158 5049 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:51:33.288258 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.288194 5049 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:51:33.288338 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.288269 5049 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:51:33.288338 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.288279 5049 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:51:33.288338 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:33.288283 5049 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-61.ec2.internal\" not found" Oct 30 12:51:33.288811 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.288794 5049 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:51:33.289682 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.289135 5049 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:45:53 +0000 UTC" deadline="2028-06-26 10:57:21.169425237 +0000 UTC" Oct 30 12:51:33.289777 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.289682 5049 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14518h5m47.879748924s" Oct 30 12:51:33.291613 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.291593 5049 factory.go:55] Registering systemd factory Oct 30 12:51:33.291835 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.291616 5049 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:51:33.291835 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.291829 5049 factory.go:153] Registering CRI-O factory Oct 30 12:51:33.291942 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.291845 5049 factory.go:223] Registration of the crio container factory successfully Oct 30 12:51:33.291942 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.291900 5049 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:51:33.291942 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.291917 5049 factory.go:103] Registering Raw factory Oct 30 12:51:33.291942 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.291931 5049 manager.go:1199] Started watching for new ooms in manager Oct 30 12:51:33.292268 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.292258 5049 manager.go:319] Starting recovery of all containers Oct 30 12:51:33.294773 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294744 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3" volumeName="kubernetes.io/projected/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-kube-api-access-qlchw" seLinuxMountContext="" Oct 30 12:51:33.294773 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294773 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caed0908-537e-4064-858e-f3f8a82b87b1" volumeName="kubernetes.io/configmap/caed0908-537e-4064-858e-f3f8a82b87b1-nginx-conf" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294783 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0471989-f151-47a5-9d39-97d85760da9e" volumeName="kubernetes.io/projected/d0471989-f151-47a5-9d39-97d85760da9e-kube-api-access-s86zc" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294792 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" volumeName="kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-bound-sa-token" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294800 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332bd965-1f4d-4749-8922-946a21e7ce75" volumeName="kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-installation-pull-secrets" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294807 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56a8078f-bd8e-42d8-86ef-4e0030e0ea76" volumeName="kubernetes.io/configmap/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-config" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294815 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9" volumeName="kubernetes.io/projected/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-kube-api-access-dm2m8" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294826 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc1fa5e9-0704-4469-83b4-bdb65f799855" volumeName="kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-trusted-ca-bundle" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294839 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc1fa5e9-0704-4469-83b4-bdb65f799855" volumeName="kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-tmp" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294851 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35925210-93d9-4a34-b733-6481da8921f3" volumeName="kubernetes.io/projected/35925210-93d9-4a34-b733-6481da8921f3-kube-api-access-npxxk" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294865 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="482cc1b2-387f-4b1e-98f5-adada095ffec" volumeName="kubernetes.io/configmap/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-config" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294878 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3" volumeName="kubernetes.io/projected/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-kube-api-access-h7t4r" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294890 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cbb909f-28ac-4b1e-9a95-5b8870886b61" volumeName="kubernetes.io/projected/0cbb909f-28ac-4b1e-9a95-5b8870886b61-kube-api-access-mpqh6" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294909 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="126669c1-ee47-49ab-a6f3-3592a66b1fd2" volumeName="kubernetes.io/configmap/126669c1-ee47-49ab-a6f3-3592a66b1fd2-serviceca" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294919 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332bd965-1f4d-4749-8922-946a21e7ce75" volumeName="kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-image-registry-private-configuration" seLinuxMountContext="" Oct 30 12:51:33.294920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294927 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0297b8f-b8e8-470e-a0cb-838697059ff4" volumeName="kubernetes.io/projected/f0297b8f-b8e8-470e-a0cb-838697059ff4-kube-api-access-5f27s" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294939 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc1fa5e9-0704-4469-83b4-bdb65f799855" volumeName="kubernetes.io/projected/fc1fa5e9-0704-4469-83b4-bdb65f799855-kube-api-access-gsmn4" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294951 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" volumeName="kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294965 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" volumeName="kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-binary-copy" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294976 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ebe4466-834b-4748-9fa5-6a01043ae3ad" volumeName="kubernetes.io/projected/8ebe4466-834b-4748-9fa5-6a01043ae3ad-kube-api-access-kjfn2" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294983 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc1fa5e9-0704-4469-83b4-bdb65f799855" volumeName="kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-snapshots" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294990 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcd977c1-bd5b-466b-a9bd-aa4889ed86b4" volumeName="kubernetes.io/projected/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-kube-api-access-g99qq" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.294998 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cbb909f-28ac-4b1e-9a95-5b8870886b61" volumeName="kubernetes.io/secret/0cbb909f-28ac-4b1e-9a95-5b8870886b61-cert" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295008 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" volumeName="kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-certificates" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295035 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" volumeName="kubernetes.io/empty-dir/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-ca-trust-extracted" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295051 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264f3a6d-7988-4ee8-bf15-bf13e991a63c" volumeName="kubernetes.io/secret/264f3a6d-7988-4ee8-bf15-bf13e991a63c-serving-cert" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295062 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58b3cad0-f9c1-4547-bb19-f1fd20041eca" volumeName="kubernetes.io/empty-dir/58b3cad0-f9c1-4547-bb19-f1fd20041eca-data-volume" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295073 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2767673-ded2-4026-8b3f-ff2582c0aede" volumeName="kubernetes.io/projected/b2767673-ded2-4026-8b3f-ff2582c0aede-kube-api-access-nx6xd" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295118 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a59c67-94e0-48e5-b8ff-28a55dea0996" volumeName="kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-daemon-config" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295132 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" volumeName="kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-trusted-ca" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295144 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332bd965-1f4d-4749-8922-946a21e7ce75" volumeName="kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-registry-certificates" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295156 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332bd965-1f4d-4749-8922-946a21e7ce75" volumeName="kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-kube-api-access-jqflv" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295167 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc1fa5e9-0704-4469-83b4-bdb65f799855" volumeName="kubernetes.io/secret/fc1fa5e9-0704-4469-83b4-bdb65f799855-serving-cert" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295179 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a59c67-94e0-48e5-b8ff-28a55dea0996" volumeName="kubernetes.io/projected/15a59c67-94e0-48e5-b8ff-28a55dea0996-kube-api-access-r774z" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295194 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="173ce15b-35dd-4a98-8f29-d13c62b56206" volumeName="kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-stats-auth" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295206 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264f3a6d-7988-4ee8-bf15-bf13e991a63c" volumeName="kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-config" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295218 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0aa96f3-47c7-48f1-98a3-dd3c4548d593" volumeName="kubernetes.io/configmap/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-konnectivity-ca" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295230 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="116675da-08c0-4101-835c-48193661ccbf" volumeName="kubernetes.io/empty-dir/116675da-08c0-4101-835c-48193661ccbf-tmp-dir" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295242 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="173ce15b-35dd-4a98-8f29-d13c62b56206" volumeName="kubernetes.io/projected/173ce15b-35dd-4a98-8f29-d13c62b56206-kube-api-access-sq4nn" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295263 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56a8078f-bd8e-42d8-86ef-4e0030e0ea76" volumeName="kubernetes.io/secret/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-serving-cert" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295276 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58b3cad0-f9c1-4547-bb19-f1fd20041eca" volumeName="kubernetes.io/projected/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-api-access-qv69b" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295285 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2767673-ded2-4026-8b3f-ff2582c0aede" volumeName="kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-script-lib" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295297 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6151a5c-6392-4151-8c74-6d518f34b4d0" volumeName="kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295309 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9" volumeName="kubernetes.io/configmap/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-cabundle" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295336 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcd977c1-bd5b-466b-a9bd-aa4889ed86b4" volumeName="kubernetes.io/configmap/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-telemetry-config" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295347 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1" volumeName="kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295358 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332bd965-1f4d-4749-8922-946a21e7ce75" volumeName="kubernetes.io/empty-dir/332bd965-1f4d-4749-8922-946a21e7ce75-ca-trust-extracted" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295367 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332bd965-1f4d-4749-8922-946a21e7ce75" volumeName="kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-bound-sa-token" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295378 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56a8078f-bd8e-42d8-86ef-4e0030e0ea76" volumeName="kubernetes.io/projected/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-kube-api-access-ndg29" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295386 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2767673-ded2-4026-8b3f-ff2582c0aede" volumeName="kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-config" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295396 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332bd965-1f4d-4749-8922-946a21e7ce75" volumeName="kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-trusted-ca" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295408 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe" volumeName="kubernetes.io/secret/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-serving-cert" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295419 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2767673-ded2-4026-8b3f-ff2582c0aede" volumeName="kubernetes.io/secret/b2767673-ded2-4026-8b3f-ff2582c0aede-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295430 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="482cc1b2-387f-4b1e-98f5-adada095ffec" volumeName="kubernetes.io/projected/482cc1b2-387f-4b1e-98f5-adada095ffec-kube-api-access-c7m2h" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295448 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f13020b-562d-470d-9bc4-70a5dd26f1e9" volumeName="kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-service-ca-bundle" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295459 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="126669c1-ee47-49ab-a6f3-3592a66b1fd2" volumeName="kubernetes.io/projected/126669c1-ee47-49ab-a6f3-3592a66b1fd2-kube-api-access-dcd62" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295475 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a59c67-94e0-48e5-b8ff-28a55dea0996" volumeName="kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-cni-binary-copy" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295487 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="173ce15b-35dd-4a98-8f29-d13c62b56206" volumeName="kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-default-certificate" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295497 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28a37fed-2af0-45d1-8e84-c658dbff6b61" volumeName="kubernetes.io/configmap/28a37fed-2af0-45d1-8e84-c658dbff6b61-iptables-alerter-script" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295509 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f13020b-562d-470d-9bc4-70a5dd26f1e9" volumeName="kubernetes.io/empty-dir/4f13020b-562d-470d-9bc4-70a5dd26f1e9-archives-path" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295520 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" volumeName="kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295532 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" volumeName="kubernetes.io/projected/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-kube-api-access-ccvtd" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295542 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f13020b-562d-470d-9bc4-70a5dd26f1e9" volumeName="kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-trusted-ca-bundle" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295553 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe" volumeName="kubernetes.io/projected/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-kube-api-access-ck6ww" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295565 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe" volumeName="kubernetes.io/configmap/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-config" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295583 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dc9e5e5-426c-4e0b-897f-54e0e28d3313" volumeName="kubernetes.io/projected/9dc9e5e5-426c-4e0b-897f-54e0e28d3313-kube-api-access-vkp4n" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295594 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2767673-ded2-4026-8b3f-ff2582c0aede" volumeName="kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-env-overrides" seLinuxMountContext="" Oct 30 12:51:33.295592 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295622 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9" volumeName="kubernetes.io/secret/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-key" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295636 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3" volumeName="kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-tmp" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295648 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="116675da-08c0-4101-835c-48193661ccbf" volumeName="kubernetes.io/configmap/116675da-08c0-4101-835c-48193661ccbf-config-volume" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295662 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" volumeName="kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-kube-api-access-s6ml9" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295672 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264f3a6d-7988-4ee8-bf15-bf13e991a63c" volumeName="kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-trusted-ca" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295686 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbb88cf5-e088-4260-82e6-ed56a2b00176" volumeName="kubernetes.io/projected/cbb88cf5-e088-4260-82e6-ed56a2b00176-kube-api-access-pc2w4" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295698 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f13020b-562d-470d-9bc4-70a5dd26f1e9" volumeName="kubernetes.io/projected/4f13020b-562d-470d-9bc4-70a5dd26f1e9-kube-api-access-4k2f6" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295708 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac" volumeName="kubernetes.io/projected/da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac-kube-api-access-tvcdx" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295721 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3" volumeName="kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-tuned" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295733 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="116675da-08c0-4101-835c-48193661ccbf" volumeName="kubernetes.io/projected/116675da-08c0-4101-835c-48193661ccbf-kube-api-access-dcdfs" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295744 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" volumeName="kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-image-registry-private-configuration" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295753 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e6304cb-862e-43ea-8688-8298f61b8c5b" volumeName="kubernetes.io/projected/6e6304cb-862e-43ea-8688-8298f61b8c5b-kube-api-access-x7x4b" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295764 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0471989-f151-47a5-9d39-97d85760da9e" volumeName="kubernetes.io/empty-dir/d0471989-f151-47a5-9d39-97d85760da9e-tmp-dir" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295773 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc1fa5e9-0704-4469-83b4-bdb65f799855" volumeName="kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-service-ca-bundle" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295785 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" volumeName="kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-installation-pull-secrets" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295797 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264f3a6d-7988-4ee8-bf15-bf13e991a63c" volumeName="kubernetes.io/projected/264f3a6d-7988-4ee8-bf15-bf13e991a63c-kube-api-access-j8xfl" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295806 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28a37fed-2af0-45d1-8e84-c658dbff6b61" volumeName="kubernetes.io/projected/28a37fed-2af0-45d1-8e84-c658dbff6b61-kube-api-access-zlxdg" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295816 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3400463b-e945-48d9-8dc5-8b8106cb6e26" volumeName="kubernetes.io/projected/3400463b-e945-48d9-8dc5-8b8106cb6e26-kube-api-access-8lvsp" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295828 5049 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0aa96f3-47c7-48f1-98a3-dd3c4548d593" volumeName="kubernetes.io/secret/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-agent-certs" seLinuxMountContext="" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295838 5049 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:51:33.298006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.295845 5049 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:51:33.311618 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.311596 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 12:51:33.312888 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.312860 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 12:51:33.325093 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.323647 5049 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:51:33.327142 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.327120 5049 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:51:33.327142 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.327145 5049 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:51:33.327263 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.327166 5049 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:51:33.327263 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:33.327208 5049 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 12:51:33.330746 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.330719 5049 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:33.332205 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.332184 5049 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 12:51:33.340461 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.340442 5049 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:33.344844 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.344728 5049 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:33.344940 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.344917 5049 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:33.346503 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.346485 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 12:51:33.346587 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.346511 5049 generic.go:356] "Generic (PLEG): container finished" podID="264f3a6d-7988-4ee8-bf15-bf13e991a63c" containerID="cf5f0644426bbd326a55d3900f2557b6ff58232881822a9651d34dd39b748bec" exitCode=255 Oct 30 12:51:33.351632 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.351579 5049 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="ab5addeb3b8ed7ad31567b27d99a41aa611606a9ec73d29ddfa8e52fdb19c7af" exitCode=0 Oct 30 12:51:33.351632 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.351603 5049 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="01852d67e6b4283f31054162adfd1b413b8c698973022763f6b9236bb2e0078b" exitCode=0 Oct 30 12:51:33.351632 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.351614 5049 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="63ff6221fb4cfdaf242f32a24d6a5b84c46ad668f3c21c523b023c9bf1fd8ce5" exitCode=0 Oct 30 12:51:33.351632 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.351624 5049 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="b6009e3868ada8cde78324dec93155308834a4dbd706e0abf43e8f308fb8aa57" exitCode=0 Oct 30 12:51:33.351816 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.351636 5049 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="9140e24858863571c58a832430da31d9c76e14ff3158e04cc4b3f231aabf42c2" exitCode=0 Oct 30 12:51:33.351816 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.351650 5049 generic.go:356] "Generic (PLEG): container finished" podID="1e0a1149-8d8b-4ae4-bf40-77a58b268d6f" containerID="539a633d742c27ad563e5d88c5cd07f85ecec9f20ecbdb9262a0281375720da9" exitCode=0 Oct 30 12:51:33.383664 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.383620 5049 generic.go:356] "Generic (PLEG): container finished" podID="9a61b1589e9c4b5de6634f4c13443710" containerID="14b11276f940ddf79840d6409eabf15fc7383af2c179412f0e14608cde55e13e" exitCode=0 Oct 30 12:51:33.394820 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.394797 5049 manager.go:324] Recovery completed Oct 30 12:51:33.416616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.416578 5049 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:51:33.416616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.416608 5049 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:51:33.416825 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.416634 5049 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:51:33.416871 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.416837 5049 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 12:51:33.416919 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.416853 5049 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 12:51:33.416919 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.416880 5049 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 12:51:33.416919 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.416888 5049 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 12:51:33.416919 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.416896 5049 policy_none.go:50] "Start" Oct 30 12:51:33.416919 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.416916 5049 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:51:33.417088 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.416929 5049 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:51:33.417920 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.417902 5049 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 12:51:33.417954 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.417923 5049 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 12:51:33.417954 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.417933 5049 policy_none.go:44] "Start" Oct 30 12:51:33.423501 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.423465 5049 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:51:33.427100 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.427086 5049 manager.go:341] "Starting Device Plugin manager" Oct 30 12:51:33.427201 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:33.427113 5049 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:51:33.427201 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.427121 5049 server.go:87] "Starting device plugin registration server" Oct 30 12:51:33.427332 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.427307 5049 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:51:33.427385 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.427342 5049 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:51:33.427448 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.427390 5049 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal"] Oct 30 12:51:33.427506 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.427492 5049 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:51:33.427617 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.427588 5049 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:51:33.427676 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.427624 5049 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:51:33.427676 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.427631 5049 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:51:33.427858 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.427786 5049 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:51:33.427614767Z","UUID":"176ce6cd-9f3f-46c2-941d-02ea4596b255","Handler":null,"Name":"","Endpoint":""} Oct 30 12:51:33.428204 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.428178 5049 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.429072 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.429037 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" event={"ID":"bac4afb01d6e1d9bf4827fcda4b8b291","Type":"ContainerStarted","Data":"ef0cc9de70cab4a13f0fc9952a6e508f66b556eb829d08381d8392749672726c"} Oct 30 12:51:33.429143 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.429078 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" event={"ID":"bac4afb01d6e1d9bf4827fcda4b8b291","Type":"ContainerStarted","Data":"f40308f148529bb5b59cc5f21ec25642c8f89bc01f8f39ecddf49e7f2dfcc3c6"} Oct 30 12:51:33.429143 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.429140 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" event={"ID":"9a61b1589e9c4b5de6634f4c13443710","Type":"ContainerStarted","Data":"4bb9e981a048a6712eaaf42af182b7e83e97955cd0f0fb66144c29e10d511f83"} Oct 30 12:51:33.429254 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.429153 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" event={"ID":"9a61b1589e9c4b5de6634f4c13443710","Type":"ContainerDied","Data":"14b11276f940ddf79840d6409eabf15fc7383af2c179412f0e14608cde55e13e"} Oct 30 12:51:33.429254 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.429166 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" event={"ID":"9a61b1589e9c4b5de6634f4c13443710","Type":"ContainerStarted","Data":"7d67e7e634aaf4db20f4c3d88b3b869f823864006342c66cd3918339f37396b7"} Oct 30 12:51:33.429384 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.429260 5049 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.429384 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:33.429291 5049 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:51:33.429733 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.429715 5049 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:51:33.429836 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.429751 5049 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:51:33.449910 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.449889 5049 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:33.450047 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:33.449942 5049 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.453469 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.453449 5049 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:33.453572 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:33.453496 5049 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.528129 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.528098 5049 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:33.532661 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.532637 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.532751 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.532667 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.532751 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.532689 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bac4afb01d6e1d9bf4827fcda4b8b291-config\") pod \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" (UID: \"bac4afb01d6e1d9bf4827fcda4b8b291\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.533658 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.533631 5049 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:33.533751 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.533667 5049 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:33.533751 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.533678 5049 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-61.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:33.533751 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.533728 5049 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.569085 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.569054 5049 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.569223 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.569136 5049 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.633195 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.633114 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.633195 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.633159 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.633195 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.633187 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bac4afb01d6e1d9bf4827fcda4b8b291-config\") pod \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" (UID: \"bac4afb01d6e1d9bf4827fcda4b8b291\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.633391 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.633226 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.633391 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.633231 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bac4afb01d6e1d9bf4827fcda4b8b291-config\") pod \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" (UID: \"bac4afb01d6e1d9bf4827fcda4b8b291\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:51:33.633391 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:33.633231 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a61b1589e9c4b5de6634f4c13443710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" (UID: \"9a61b1589e9c4b5de6634f4c13443710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:51:34.274688 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.274657 5049 apiserver.go:52] "Watching apiserver" Oct 30 12:51:34.314731 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.314706 5049 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:51:34.315418 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.315391 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh","openshift-multus/multus-s78dq","openshift-dns/dns-default-94rws","kube-system/konnectivity-agent-cdptr","kube-system/global-pull-secret-syncer-jtc6z","kube-system/kas-connection-checker-685895c7c9-crmm8","openshift-service-ca/service-ca-5fffc97bdf-xzkxq","openshift-image-registry/image-registry-87dcb74f4-755j7","openshift-image-registry/node-ca-6p76v","openshift-insights/periodic-gathering-7mcfh-fnbft","openshift-network-console/networking-console-plugin-86c478bb88-f4hf7","openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww","openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm","openshift-network-operator/iptables-alerter-n4rct","openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv","openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd","openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw","openshift-network-diagnostics/network-check-target-b9v2t","openshift-image-registry/image-registry-85cb969f47-rcssw","openshift-ingress-canary/ingress-canary-pzn2g","openshift-ovn-kubernetes/ovnkube-node-gp5jc","kube-system/kas-connection-checker-685895c7c9-97qv2","openshift-dns/node-resolver-v4l8n","openshift-ingress/router-default-df858cf77-mnl8f","openshift-insights/insights-runtime-extractor-bzlvd","openshift-multus/multus-additional-cni-plugins-dnwpp","kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx","openshift-insights/insights-operator-7c4986d449-6qdnl","kube-system/kas-connection-checker-685895c7c9-bqzs4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal","openshift-multus/network-metrics-daemon-l7pxj","openshift-console-operator/console-operator-7cd56b8dbc-l2xkr","openshift-cluster-node-tuning-operator/tuned-2ktf5"] Oct 30 12:51:34.316638 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.316622 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.316996 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.316976 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:34.317550 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.317532 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.319070 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.319050 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:34.320394 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.320379 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:34.324883 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.324844 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:34.328914 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.328894 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.329017 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.328940 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.329146 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.329128 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.329503 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.329487 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94rws" Oct 30 12:51:34.329564 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.329543 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 12:51:34.329624 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.329574 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 12:51:34.329787 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.329760 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 12:51:34.330794 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.330780 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.335898 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.335881 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 12:51:34.335898 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.335888 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.336054 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.336040 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.349715 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.349699 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.349962 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.349949 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:51:34.351350 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.351313 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 12:51:34.357466 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.357452 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 12:51:34.357521 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.357475 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 12:51:34.357697 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.357683 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h5w6k\"" Oct 30 12:51:34.363686 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.363672 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:51:34.365546 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.365448 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:51:34.365614 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.365584 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hfgh7\"" Oct 30 12:51:34.365666 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.365631 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:51:34.365883 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.365867 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 12:51:34.365951 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.365936 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 12:51:34.366217 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366198 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-z66zl\"" Oct 30 12:51:34.366300 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366233 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mt9m7\"" Oct 30 12:51:34.366300 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366272 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.366460 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366444 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb6s2\"" Oct 30 12:51:34.366535 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366458 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.366571 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366531 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d8ng4\"" Oct 30 12:51:34.366571 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366467 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rnjq7\"" Oct 30 12:51:34.366571 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366541 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g6csn\"" Oct 30 12:51:34.366571 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366480 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.366571 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366486 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.366739 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366590 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:51:34.366739 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366598 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:51:34.366739 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366606 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-24gr7\"" Oct 30 12:51:34.366739 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366641 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366740 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fv4k6\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366755 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366776 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366788 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366794 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366813 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366821 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366829 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366841 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366845 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366846 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nklv7\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366829 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wb7zr\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366856 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366873 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 12:51:34.366912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366874 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366940 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366950 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.366962 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367165 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367182 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367191 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367199 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367205 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367211 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r9ldj\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367232 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367280 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367292 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367404 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367485 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367519 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.367591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367577 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:51:34.368156 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.367642 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:51:34.368258 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.368237 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.368434 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.368417 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-588rd\"" Oct 30 12:51:34.368668 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.368653 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:51:34.368725 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.368695 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.368795 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.368778 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.368883 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.368869 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:51:34.368950 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.368936 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:51:34.369045 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.369031 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.369297 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.369279 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.369399 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.369383 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 12:51:34.369490 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.369474 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.369657 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.369641 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 12:51:34.369843 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.369825 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:51:34.369892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.369826 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:51:34.370003 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.369986 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:51:34.370155 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370142 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:51:34.370213 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370171 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vr75\"" Oct 30 12:51:34.370260 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370214 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rq622\"" Oct 30 12:51:34.370416 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370400 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 12:51:34.370473 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370423 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 12:51:34.370473 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370428 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 12:51:34.370545 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370503 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.370591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370543 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:51:34.370591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370563 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 12:51:34.370725 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370708 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 12:51:34.370775 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370729 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.370775 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370715 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhkwq\"" Oct 30 12:51:34.370775 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370759 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:34.370900 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.370843 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:51:34.375579 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.375556 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 12:51:34.376706 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.376686 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 12:51:34.376821 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.376735 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 12:51:34.387061 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.387044 5049 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:51:34.387147 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.387100 5049 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:51:34.390967 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.390954 5049 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:51:34.411725 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.411700 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 12:51:34.412250 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.412224 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 12:51:34.412380 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.412257 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 12:51:34.412988 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.412975 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:51:34.413529 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.413512 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9fz8p\"" Oct 30 12:51:34.413529 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.413513 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-42xpg\"" Oct 30 12:51:34.413634 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.413537 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pdl9w\"" Oct 30 12:51:34.413634 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.413537 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4vf72\"" Oct 30 12:51:34.413922 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.413781 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 12:51:34.413922 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.413792 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5jsfn\"" Oct 30 12:51:34.413922 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.413796 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-78z4l\"" Oct 30 12:51:34.413922 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.413810 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kq2jq\"" Oct 30 12:51:34.413922 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.413812 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4g9fn\"" Oct 30 12:51:34.414080 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.413999 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n54s8\"" Oct 30 12:51:34.434353 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.434316 5049 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:34.434432 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.434382 5049 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-61.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-61.ec2.internal" Oct 30 12:51:34.436624 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436603 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:34.436665 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436637 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcdx\" (UniqueName: \"kubernetes.io/projected/da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac-kube-api-access-tvcdx\") pod \"migrator-66df5d496b-jntww\" (UID: \"da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww" Oct 30 12:51:34.436665 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436654 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116675da-08c0-4101-835c-48193661ccbf-tmp-dir\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:34.436741 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436672 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-env-overrides\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.436741 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436714 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-kubernetes\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.436814 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436747 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-var-lib-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.436814 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436772 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116675da-08c0-4101-835c-48193661ccbf-tmp-dir\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:34.436814 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436776 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-trusted-ca-bundle\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:34.436950 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436827 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndg29\" (UniqueName: \"kubernetes.io/projected/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-kube-api-access-ndg29\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:34.436950 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436902 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-ca-trust-extracted\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.436950 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436943 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysconfig\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.437051 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436976 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-ca-trust-extracted\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.437051 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.436991 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-k8s-cni-cncf-io\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.437051 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437013 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-env-overrides\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.437051 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437032 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-sys-fs\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.437266 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437098 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xfl\" (UniqueName: \"kubernetes.io/projected/264f3a6d-7988-4ee8-bf15-bf13e991a63c-kube-api-access-j8xfl\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:34.437266 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437130 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/332bd965-1f4d-4749-8922-946a21e7ce75-ca-trust-extracted\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.437266 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437148 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-trusted-ca-bundle\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:34.437266 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437160 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-bin\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.437266 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437200 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/116675da-08c0-4101-835c-48193661ccbf-config-volume\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:34.437266 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437214 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/332bd965-1f4d-4749-8922-946a21e7ce75-ca-trust-extracted\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.437266 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437245 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-var-lib-kubelet\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.437563 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437277 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2767673-ded2-4026-8b3f-ff2582c0aede-ovn-node-metrics-cert\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.437563 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437346 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.437563 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437362 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.437563 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437379 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-key\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:34.437563 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437398 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmn4\" (UniqueName: \"kubernetes.io/projected/fc1fa5e9-0704-4469-83b4-bdb65f799855-kube-api-access-gsmn4\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.437563 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437405 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/116675da-08c0-4101-835c-48193661ccbf-config-volume\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:34.437563 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437413 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-slash\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.437563 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437486 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-netd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.437563 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437528 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-image-registry-private-configuration\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437583 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-netns\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437632 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-key\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437661 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-stats-auth\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437695 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-etc-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437698 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2767673-ded2-4026-8b3f-ff2582c0aede-ovn-node-metrics-cert\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437754 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-image-registry-private-configuration\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437810 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437846 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxdg\" (UniqueName: \"kubernetes.io/projected/28a37fed-2af0-45d1-8e84-c658dbff6b61-kube-api-access-zlxdg\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437871 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437874 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-stats-auth\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437897 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.437960 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437917 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a37fed-2af0-45d1-8e84-c658dbff6b61-host-slash\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.437975 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-trusted-ca\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438029 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438061 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-agent-certs\") pod \"konnectivity-agent-cdptr\" (UID: \"b0aa96f3-47c7-48f1-98a3-dd3c4548d593\") " pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438089 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-snapshots\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438116 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvtd\" (UniqueName: \"kubernetes.io/projected/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-kube-api-access-ccvtd\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438144 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438172 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438192 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-snapshots\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438222 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-cni-binary-copy\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438254 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438282 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-certificates\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438292 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438314 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58b3cad0-f9c1-4547-bb19-f1fd20041eca-crio-socket\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438348 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-trusted-ca\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438358 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438386 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-sys\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438396 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-agent-certs\") pod \"konnectivity-agent-cdptr\" (UID: \"b0aa96f3-47c7-48f1-98a3-dd3c4548d593\") " pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438411 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438462 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k2f6\" (UniqueName: \"kubernetes.io/projected/4f13020b-562d-470d-9bc4-70a5dd26f1e9-kube-api-access-4k2f6\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438467 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-certificates\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438490 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-tuned\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.438557 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438553 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-cni-binary-copy\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438568 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438598 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-config\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438620 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-tuned\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438628 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd62\" (UniqueName: \"kubernetes.io/projected/126669c1-ee47-49ab-a6f3-3592a66b1fd2-kube-api-access-dcd62\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438633 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438663 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438747 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqh6\" (UniqueName: \"kubernetes.io/projected/0cbb909f-28ac-4b1e-9a95-5b8870886b61-kube-api-access-mpqh6\") pod \"ingress-canary-pzn2g\" (UID: \"0cbb909f-28ac-4b1e-9a95-5b8870886b61\") " pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438769 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-config\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438778 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-kubelet\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438815 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-systemd-units\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438832 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-service-ca-bundle\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438852 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-socket-dir-parent\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438899 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438907 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-kubelet\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438940 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-conf-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438962 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2m8\" (UniqueName: \"kubernetes.io/projected/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-kube-api-access-dm2m8\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.438978 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-system-cni-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439018 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439043 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58b3cad0-f9c1-4547-bb19-f1fd20041eca-data-volume\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439104 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-installation-pull-secrets\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439141 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58b3cad0-f9c1-4547-bb19-f1fd20041eca-data-volume\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439188 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-config\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439279 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvsp\" (UniqueName: \"kubernetes.io/projected/3400463b-e945-48d9-8dc5-8b8106cb6e26-kube-api-access-8lvsp\") pod \"kas-connection-checker-685895c7c9-bqzs4\" (UID: \"3400463b-e945-48d9-8dc5-8b8106cb6e26\") " pod="kube-system/kas-connection-checker-685895c7c9-bqzs4" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439309 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-cnibin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439347 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-konnectivity-ca\") pod \"konnectivity-agent-cdptr\" (UID: \"b0aa96f3-47c7-48f1-98a3-dd3c4548d593\") " pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439353 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-service-ca-bundle\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439372 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-installation-pull-secrets\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439414 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439443 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-log-socket\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439463 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-config\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439513 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439544 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6ww\" (UniqueName: \"kubernetes.io/projected/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-kube-api-access-ck6ww\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439555 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0aa96f3-47c7-48f1-98a3-dd3c4548d593-konnectivity-ca\") pod \"konnectivity-agent-cdptr\" (UID: \"b0aa96f3-47c7-48f1-98a3-dd3c4548d593\") " pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:51:34.439578 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439588 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-registration-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439673 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-service-ca-bundle\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439706 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-ovn\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439736 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28a37fed-2af0-45d1-8e84-c658dbff6b61-iptables-alerter-script\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439773 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-tmp\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439798 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-default-certificate\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439897 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1fa5e9-0704-4469-83b4-bdb65f799855-serving-cert\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439909 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-tmp\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.439943 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/126669c1-ee47-49ab-a6f3-3592a66b1fd2-host\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440031 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-default-certificate\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440036 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28a37fed-2af0-45d1-8e84-c658dbff6b61-iptables-alerter-script\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440049 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g99qq\" (UniqueName: \"kubernetes.io/projected/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-kube-api-access-g99qq\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440080 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcdfs\" (UniqueName: \"kubernetes.io/projected/116675da-08c0-4101-835c-48193661ccbf-kube-api-access-dcdfs\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440108 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7m2h\" (UniqueName: \"kubernetes.io/projected/482cc1b2-387f-4b1e-98f5-adada095ffec-kube-api-access-c7m2h\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440136 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-lib-modules\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440165 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-multus-certs\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440194 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-tmp\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440221 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-config\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440225 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fa5e9-0704-4469-83b4-bdb65f799855-service-ca-bundle\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440356 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440360 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1fa5e9-0704-4469-83b4-bdb65f799855-tmp\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440382 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1fa5e9-0704-4469-83b4-bdb65f799855-serving-cert\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440402 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlchw\" (UniqueName: \"kubernetes.io/projected/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-kube-api-access-qlchw\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440424 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7x4b\" (UniqueName: \"kubernetes.io/projected/6e6304cb-862e-43ea-8688-8298f61b8c5b-kube-api-access-x7x4b\") pod \"network-check-source-6f78d59f99-68gnd\" (UID: \"6e6304cb-862e-43ea-8688-8298f61b8c5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440428 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-config\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440442 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-trusted-ca\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440466 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440481 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-conf\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440498 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f27s\" (UniqueName: \"kubernetes.io/projected/f0297b8f-b8e8-470e-a0cb-838697059ff4-kube-api-access-5f27s\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440517 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx6xd\" (UniqueName: \"kubernetes.io/projected/b2767673-ded2-4026-8b3f-ff2582c0aede-kube-api-access-nx6xd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440691 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-etc-kubernetes\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440711 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-trusted-ca\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440715 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440761 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-socket-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440783 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-registry-certificates\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440798 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-dbus\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440816 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.440857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440847 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-kubelet-config\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440905 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-original-pull-secret\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440908 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-registry-certificates\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440933 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.440966 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc2w4\" (UniqueName: \"kubernetes.io/projected/cbb88cf5-e088-4260-82e6-ed56a2b00176-kube-api-access-pc2w4\") pod \"kas-connection-checker-685895c7c9-crmm8\" (UID: \"cbb88cf5-e088-4260-82e6-ed56a2b00176\") " pod="kube-system/kas-connection-checker-685895c7c9-crmm8" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441002 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-netns\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441021 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-node-log\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441035 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441040 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-serving-cert\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441056 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264f3a6d-7988-4ee8-bf15-bf13e991a63c-serving-cert\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441139 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caed0908-537e-4064-858e-f3f8a82b87b1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441156 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-trusted-ca\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441172 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-run\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441185 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-serving-cert\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441188 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7t4r\" (UniqueName: \"kubernetes.io/projected/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-kube-api-access-h7t4r\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441226 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/264f3a6d-7988-4ee8-bf15-bf13e991a63c-serving-cert\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441304 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r774z\" (UniqueName: \"kubernetes.io/projected/15a59c67-94e0-48e5-b8ff-28a55dea0996-kube-api-access-r774z\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441348 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-device-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441374 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfn2\" (UniqueName: \"kubernetes.io/projected/8ebe4466-834b-4748-9fa5-6a01043ae3ad-kube-api-access-kjfn2\") pod \"migrator-66df5d496b-6dhsv\" (UID: \"8ebe4466-834b-4748-9fa5-6a01043ae3ad\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441394 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-bound-sa-token\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441400 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caed0908-537e-4064-858e-f3f8a82b87b1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441502 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/264f3a6d-7988-4ee8-bf15-bf13e991a63c-trusted-ca\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441565 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqflv\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-kube-api-access-jqflv\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441591 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-modprobe-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441606 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441629 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-host\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441666 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-bin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441694 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-cabundle\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441718 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cnibin\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441761 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441807 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ml9\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-kube-api-access-s6ml9\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441826 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxxk\" (UniqueName: \"kubernetes.io/projected/35925210-93d9-4a34-b733-6481da8921f3-kube-api-access-npxxk\") pod \"kas-connection-checker-685895c7c9-97qv2\" (UID: \"35925210-93d9-4a34-b733-6481da8921f3\") " pod="kube-system/kas-connection-checker-685895c7c9-97qv2" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441842 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-daemon-config\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441858 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/126669c1-ee47-49ab-a6f3-3592a66b1fd2-serviceca\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441874 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkp4n\" (UniqueName: \"kubernetes.io/projected/9dc9e5e5-426c-4e0b-897f-54e0e28d3313-kube-api-access-vkp4n\") pod \"volume-data-source-validator-58c9f5f46d-vf8sw\" (UID: \"9dc9e5e5-426c-4e0b-897f-54e0e28d3313\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441898 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-signing-cabundle\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441898 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441939 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441950 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-script-lib\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.441976 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f13020b-562d-470d-9bc4-70a5dd26f1e9-archives-path\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442005 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442044 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442077 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-daemon-config\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442086 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f13020b-562d-470d-9bc4-70a5dd26f1e9-archives-path\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442080 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-multus\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442117 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/126669c1-ee47-49ab-a6f3-3592a66b1fd2-serviceca\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442120 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-hostroot\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442146 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0471989-f151-47a5-9d39-97d85760da9e-hosts-file\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442185 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-image-registry-private-configuration\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442203 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-systemd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442224 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.442212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442236 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2767673-ded2-4026-8b3f-ff2582c0aede-ovnkube-script-lib\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442254 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq4nn\" (UniqueName: \"kubernetes.io/projected/173ce15b-35dd-4a98-8f29-d13c62b56206-kube-api-access-sq4nn\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442267 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442288 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-system-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442336 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442358 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86zc\" (UniqueName: \"kubernetes.io/projected/d0471989-f151-47a5-9d39-97d85760da9e-kube-api-access-s86zc\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442381 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-image-registry-private-configuration\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442389 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-installation-pull-secrets\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442406 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-bound-sa-token\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442507 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442588 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-systemd\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442618 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0471989-f151-47a5-9d39-97d85760da9e-tmp-dir\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442640 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cbb909f-28ac-4b1e-9a95-5b8870886b61-cert\") pod \"ingress-canary-pzn2g\" (UID: \"0cbb909f-28ac-4b1e-9a95-5b8870886b61\") " pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442667 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442712 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-installation-pull-secrets\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442760 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0471989-f151-47a5-9d39-97d85760da9e-tmp-dir\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442777 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv69b\" (UniqueName: \"kubernetes.io/projected/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-api-access-qv69b\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.443616 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.442816 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cbb909f-28ac-4b1e-9a95-5b8870886b61-cert\") pod \"ingress-canary-pzn2g\" (UID: \"0cbb909f-28ac-4b1e-9a95-5b8870886b61\") " pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:34.457577 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.457557 5049 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:34.457675 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.457610 5049 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-61.ec2.internal" Oct 30 12:51:34.466892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.466872 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r774z\" (UniqueName: \"kubernetes.io/projected/15a59c67-94e0-48e5-b8ff-28a55dea0996-kube-api-access-r774z\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.468696 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.468680 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-bound-sa-token\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.469501 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.469479 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcdx\" (UniqueName: \"kubernetes.io/projected/da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac-kube-api-access-tvcdx\") pod \"migrator-66df5d496b-jntww\" (UID: \"da9ebb2a-bb7e-4fbd-b944-b2347cfff6ac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jntww" Oct 30 12:51:34.469786 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.469770 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlchw\" (UniqueName: \"kubernetes.io/projected/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-kube-api-access-qlchw\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:34.469995 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.469980 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndg29\" (UniqueName: \"kubernetes.io/projected/56a8078f-bd8e-42d8-86ef-4e0030e0ea76-kube-api-access-ndg29\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv\" (UID: \"56a8078f-bd8e-42d8-86ef-4e0030e0ea76\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" Oct 30 12:51:34.470267 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.470253 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcdfs\" (UniqueName: \"kubernetes.io/projected/116675da-08c0-4101-835c-48193661ccbf-kube-api-access-dcdfs\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:34.470984 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.470968 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvtd\" (UniqueName: \"kubernetes.io/projected/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-kube-api-access-ccvtd\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.475008 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.474990 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k2f6\" (UniqueName: \"kubernetes.io/projected/4f13020b-562d-470d-9bc4-70a5dd26f1e9-kube-api-access-4k2f6\") pod \"periodic-gathering-7mcfh-fnbft\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:51:34.492209 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.492187 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc2w4\" (UniqueName: \"kubernetes.io/projected/cbb88cf5-e088-4260-82e6-ed56a2b00176-kube-api-access-pc2w4\") pod \"kas-connection-checker-685895c7c9-crmm8\" (UID: \"cbb88cf5-e088-4260-82e6-ed56a2b00176\") " pod="kube-system/kas-connection-checker-685895c7c9-crmm8" Oct 30 12:51:34.493892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.493872 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7m2h\" (UniqueName: \"kubernetes.io/projected/482cc1b2-387f-4b1e-98f5-adada095ffec-kube-api-access-c7m2h\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:34.497524 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.497507 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxxk\" (UniqueName: \"kubernetes.io/projected/35925210-93d9-4a34-b733-6481da8921f3-kube-api-access-npxxk\") pod \"kas-connection-checker-685895c7c9-97qv2\" (UID: \"35925210-93d9-4a34-b733-6481da8921f3\") " pod="kube-system/kas-connection-checker-685895c7c9-97qv2" Oct 30 12:51:34.497776 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.497761 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm47z\" (UniqueName: \"kubernetes.io/projected/b6151a5c-6392-4151-8c74-6d518f34b4d0-kube-api-access-pm47z\") pod \"network-check-target-b9v2t\" (UID: \"b6151a5c-6392-4151-8c74-6d518f34b4d0\") " pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:34.499680 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.499665 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqflv\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-kube-api-access-jqflv\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.499871 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.499854 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f27s\" (UniqueName: \"kubernetes.io/projected/f0297b8f-b8e8-470e-a0cb-838697059ff4-kube-api-access-5f27s\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.515212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.515184 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq4nn\" (UniqueName: \"kubernetes.io/projected/173ce15b-35dd-4a98-8f29-d13c62b56206-kube-api-access-sq4nn\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.515390 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.515373 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmn4\" (UniqueName: \"kubernetes.io/projected/fc1fa5e9-0704-4469-83b4-bdb65f799855-kube-api-access-gsmn4\") pod \"insights-operator-7c4986d449-6qdnl\" (UID: \"fc1fa5e9-0704-4469-83b4-bdb65f799855\") " pod="openshift-insights/insights-operator-7c4986d449-6qdnl" Oct 30 12:51:34.515390 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.515382 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7t4r\" (UniqueName: \"kubernetes.io/projected/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-kube-api-access-h7t4r\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.515591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.515373 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv69b\" (UniqueName: \"kubernetes.io/projected/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-api-access-qv69b\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.515591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.515429 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xfl\" (UniqueName: \"kubernetes.io/projected/264f3a6d-7988-4ee8-bf15-bf13e991a63c-kube-api-access-j8xfl\") pod \"console-operator-7cd56b8dbc-l2xkr\" (UID: \"264f3a6d-7988-4ee8-bf15-bf13e991a63c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:34.515591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.515490 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxdg\" (UniqueName: \"kubernetes.io/projected/28a37fed-2af0-45d1-8e84-c658dbff6b61-kube-api-access-zlxdg\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:51:34.515591 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.515549 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ml9\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-kube-api-access-s6ml9\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.515866 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.515615 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86zc\" (UniqueName: \"kubernetes.io/projected/d0471989-f151-47a5-9d39-97d85760da9e-kube-api-access-s86zc\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:51:34.515939 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.515917 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvsp\" (UniqueName: \"kubernetes.io/projected/3400463b-e945-48d9-8dc5-8b8106cb6e26-kube-api-access-8lvsp\") pod \"kas-connection-checker-685895c7c9-bqzs4\" (UID: \"3400463b-e945-48d9-8dc5-8b8106cb6e26\") " pod="kube-system/kas-connection-checker-685895c7c9-bqzs4" Oct 30 12:51:34.516059 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.516015 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcd62\" (UniqueName: \"kubernetes.io/projected/126669c1-ee47-49ab-a6f3-3592a66b1fd2-kube-api-access-dcd62\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:51:34.516059 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.516015 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx6xd\" (UniqueName: \"kubernetes.io/projected/b2767673-ded2-4026-8b3f-ff2582c0aede-kube-api-access-nx6xd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.516195 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.516083 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfn2\" (UniqueName: \"kubernetes.io/projected/8ebe4466-834b-4748-9fa5-6a01043ae3ad-kube-api-access-kjfn2\") pod \"migrator-66df5d496b-6dhsv\" (UID: \"8ebe4466-834b-4748-9fa5-6a01043ae3ad\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dhsv" Oct 30 12:51:34.516195 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.516133 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g99qq\" (UniqueName: \"kubernetes.io/projected/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-kube-api-access-g99qq\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:34.516462 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.516440 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7x4b\" (UniqueName: \"kubernetes.io/projected/6e6304cb-862e-43ea-8688-8298f61b8c5b-kube-api-access-x7x4b\") pod \"network-check-source-6f78d59f99-68gnd\" (UID: \"6e6304cb-862e-43ea-8688-8298f61b8c5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-68gnd" Oct 30 12:51:34.516655 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.516636 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2m8\" (UniqueName: \"kubernetes.io/projected/ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9-kube-api-access-dm2m8\") pod \"service-ca-5fffc97bdf-xzkxq\" (UID: \"ee31a353-85b2-44ed-a5ed-5b6dbd9dc7b9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xzkxq" Oct 30 12:51:34.516738 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.516719 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqh6\" (UniqueName: \"kubernetes.io/projected/0cbb909f-28ac-4b1e-9a95-5b8870886b61-kube-api-access-mpqh6\") pod \"ingress-canary-pzn2g\" (UID: \"0cbb909f-28ac-4b1e-9a95-5b8870886b61\") " pod="openshift-ingress-canary/ingress-canary-pzn2g" Oct 30 12:51:34.516792 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.516766 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkp4n\" (UniqueName: \"kubernetes.io/projected/9dc9e5e5-426c-4e0b-897f-54e0e28d3313-kube-api-access-vkp4n\") pod \"volume-data-source-validator-58c9f5f46d-vf8sw\" (UID: \"9dc9e5e5-426c-4e0b-897f-54e0e28d3313\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vf8sw" Oct 30 12:51:34.516915 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.516900 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6ww\" (UniqueName: \"kubernetes.io/projected/6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe-kube-api-access-ck6ww\") pod \"service-ca-operator-544d749c8c-45lpn\" (UID: \"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" Oct 30 12:51:34.516946 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.516907 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-bound-sa-token\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.523791 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.523772 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:34.543162 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543097 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-netd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.543162 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543128 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-netns\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.543162 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543144 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-etc-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.543162 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543157 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.543428 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543207 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-etc-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.543428 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543215 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-netd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.543428 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543219 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-netns\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.543428 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543274 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.543428 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543302 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a37fed-2af0-45d1-8e84-c658dbff6b61-host-slash\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:51:34.543428 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543359 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28a37fed-2af0-45d1-8e84-c658dbff6b61-host-slash\") pod \"iptables-alerter-n4rct\" (UID: \"28a37fed-2af0-45d1-8e84-c658dbff6b61\") " pod="openshift-network-operator/iptables-alerter-n4rct" Oct 30 12:51:34.543428 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543404 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:34.543428 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543424 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58b3cad0-f9c1-4547-bb19-f1fd20041eca-crio-socket\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543441 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543455 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-sys\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543471 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543520 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543529 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58b3cad0-f9c1-4547-bb19-f1fd20041eca-crio-socket\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543545 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-sys\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.543589 5049 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543607 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-kubelet\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543655 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-kubelet\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543622 5049 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.543671 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:35.043648538 +0000 UTC m=+1.858842962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543702 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-systemd-units\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543729 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-socket-dir-parent\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543751 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-kubelet\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543772 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-conf-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543805 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-system-cni-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543829 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-socket-dir-parent\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.543848 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543831 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-cnibin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543860 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-kubelet\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543868 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-log-socket\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543860 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-conf-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543885 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-cnibin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543898 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-systemd-units\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543901 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543918 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-system-cni-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543923 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-log-socket\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543936 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-registration-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543949 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-multus-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543957 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-ovn\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.543975 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-ovn\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544011 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/126669c1-ee47-49ab-a6f3-3592a66b1fd2-host\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544025 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-registration-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544030 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-lib-modules\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544050 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-multus-certs\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544064 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/126669c1-ee47-49ab-a6f3-3592a66b1fd2-host\") pod \"node-ca-6p76v\" (UID: \"126669c1-ee47-49ab-a6f3-3592a66b1fd2\") " pod="openshift-image-registry/node-ca-6p76v" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544070 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544096 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544119 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-conf\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544145 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-etc-kubernetes\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544157 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544162 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-lib-modules\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544168 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-socket-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544192 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-dbus\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544209 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-etc-kubernetes\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544212 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-kubelet-config\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544235 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544275 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-netns\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544282 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-conf\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544297 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-node-log\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544336 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-run\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544353 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-kubelet-config\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544368 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-device-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544396 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-modprobe-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544407 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-socket-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544420 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544443 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-host\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544467 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-bin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544495 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cnibin\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544515 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544527 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544554 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544581 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-multus\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544587 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-node-log\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544613 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-hostroot\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544627 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-multus\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544640 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0471989-f151-47a5-9d39-97d85760da9e-hosts-file\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544692 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-run\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544731 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-modprobe-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.544720 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544731 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-device-dir\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544767 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0471989-f151-47a5-9d39-97d85760da9e-hosts-file\") pod \"node-resolver-v4l8n\" (UID: \"d0471989-f151-47a5-9d39-97d85760da9e\") " pod="openshift-dns/node-resolver-v4l8n" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544778 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e0a1149-8d8b-4ae4-bf40-77a58b268d6f-cnibin\") pod \"multus-additional-cni-plugins-dnwpp\" (UID: \"1e0a1149-8d8b-4ae4-bf40-77a58b268d6f\") " pod="openshift-multus/multus-additional-cni-plugins-dnwpp" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544786 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysctl-d\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544799 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-systemd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544812 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-hostroot\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544822 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58b3cad0-f9c1-4547-bb19-f1fd20041eca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544831 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-run-systemd\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544840 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544855 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544555 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-run-netns\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544877 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-var-lib-cni-bin\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544458 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1-dbus\") pod \"global-pull-secret-syncer-jtc6z\" (UID: \"00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1\") " pod="kube-system/global-pull-secret-syncer-jtc6z" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544876 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-system-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544914 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-host\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544915 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544927 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544947 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544972 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-systemd\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.544983 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-system-cni-dir\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.545006 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:35.04498876 +0000 UTC m=+1.860183179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : configmap references non-existent config key: service-ca.crt Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545026 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545030 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-systemd\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.545047 5049 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545052 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545081 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-kubernetes\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.545092 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls podName:482cc1b2-387f-4b1e-98f5-adada095ffec nodeName:}" failed. No retries permitted until 2026-10-30 12:51:35.045080626 +0000 UTC m=+1.860275041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jsqvh" (UID: "482cc1b2-387f-4b1e-98f5-adada095ffec") : secret "samples-operator-tls" not found Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.545117 5049 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545119 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-var-lib-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545123 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-kubernetes\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545148 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysconfig\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.545172 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls podName:fcd977c1-bd5b-466b-a9bd-aa4889ed86b4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:35.045157273 +0000 UTC m=+1.860351690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlskm" (UID: "fcd977c1-bd5b-466b-a9bd-aa4889ed86b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.545187 5049 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545192 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-k8s-cni-cncf-io\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545196 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-etc-sysconfig\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.545218 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls podName:116675da-08c0-4101-835c-48193661ccbf nodeName:}" failed. No retries permitted until 2026-10-30 12:51:35.045208427 +0000 UTC m=+1.860402851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls") pod "dns-default-94rws" (UID: "116675da-08c0-4101-835c-48193661ccbf") : secret "dns-default-metrics-tls" not found Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545235 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-k8s-cni-cncf-io\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545244 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-var-lib-openvswitch\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545235 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-sys-fs\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545274 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-sys-fs\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545281 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-bin\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545305 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-var-lib-kubelet\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545334 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15a59c67-94e0-48e5-b8ff-28a55dea0996-host-run-multus-certs\") pod \"multus-s78dq\" (UID: \"15a59c67-94e0-48e5-b8ff-28a55dea0996\") " pod="openshift-multus/multus-s78dq" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545360 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545379 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-cni-bin\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545388 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545382 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3-var-lib-kubelet\") pod \"tuned-2ktf5\" (UID: \"02b70bdc-18a4-4b6f-89c3-2ed5dc1e8bb3\") " pod="openshift-cluster-node-tuning-operator/tuned-2ktf5" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545417 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-slash\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545448 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0297b8f-b8e8-470e-a0cb-838697059ff4-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzqcx\" (UID: \"f0297b8f-b8e8-470e-a0cb-838697059ff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqcx" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.545451 5049 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.545486 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2767673-ded2-4026-8b3f-ff2582c0aede-host-slash\") pod \"ovnkube-node-gp5jc\" (UID: \"b2767673-ded2-4026-8b3f-ff2582c0aede\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:34.545495 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:35.045485386 +0000 UTC m=+1.860679813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : secret "router-metrics-certs-default" not found Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.546360 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caed0908-537e-4064-858e-f3f8a82b87b1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-f4hf7\" (UID: \"caed0908-537e-4064-858e-f3f8a82b87b1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.546504 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") pod \"image-registry-85cb969f47-rcssw\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.546892 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.546548 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") pod \"image-registry-87dcb74f4-755j7\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.548457 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.547007 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3-metrics-certs\") pod \"network-metrics-daemon-l7pxj\" (UID: \"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3\") " pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:34.618183 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.618158 5049 scope.go:122] "RemoveContainer" containerID="cf5f0644426bbd326a55d3900f2557b6ff58232881822a9651d34dd39b748bec" Oct 30 12:51:34.618183 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.618183 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:34.620625 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.620598 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" Oct 30 12:51:34.626239 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.626221 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7pxj" Oct 30 12:51:34.629554 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.629534 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:34.891525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.891489 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:34.976479 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.976444 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-87dcb74f4-755j7"] Oct 30 12:51:34.980879 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:34.980843 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d13dd2f_93e1_42f7_a07c_d7e0ebf83e88.slice/crio-ee00a8471e20afcee9cc6f824850213d5bc33bdb36fd9da2ebfcb5261b6fd56e WatchSource:0}: Error finding container ee00a8471e20afcee9cc6f824850213d5bc33bdb36fd9da2ebfcb5261b6fd56e: Status 404 returned error can't find the container with id ee00a8471e20afcee9cc6f824850213d5bc33bdb36fd9da2ebfcb5261b6fd56e Oct 30 12:51:34.991964 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.991891 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-f4hf7"] Oct 30 12:51:34.995467 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:34.995407 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaed0908_537e_4064_858e_f3f8a82b87b1.slice/crio-478b10c07789e1aef7f0549198e7b224788e45dd07a3d247b008f953bec7907f WatchSource:0}: Error finding container 478b10c07789e1aef7f0549198e7b224788e45dd07a3d247b008f953bec7907f: Status 404 returned error can't find the container with id 478b10c07789e1aef7f0549198e7b224788e45dd07a3d247b008f953bec7907f Oct 30 12:51:34.997655 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:34.997639 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:51:35.026861 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.026665 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85cb969f47-rcssw"] Oct 30 12:51:35.029703 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:35.029681 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332bd965_1f4d_4749_8922_946a21e7ce75.slice/crio-aee9e491c329d03d8788bc9356fd1b51d5076ca4ade3155990f1de2ee37d4262 WatchSource:0}: Error finding container aee9e491c329d03d8788bc9356fd1b51d5076ca4ade3155990f1de2ee37d4262: Status 404 returned error can't find the container with id aee9e491c329d03d8788bc9356fd1b51d5076ca4ade3155990f1de2ee37d4262 Oct 30 12:51:35.050278 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.050256 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:35.050403 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.050296 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:35.050403 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.050364 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:35.050403 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.050384 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:35.050403 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.050401 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:35.050583 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050422 5049 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:51:35.050583 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050479 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:36.050465576 +0000 UTC m=+2.865659977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : secret "router-metrics-certs-default" not found Oct 30 12:51:35.050583 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050487 5049 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:35.050583 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.050417 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:35.050583 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050519 5049 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:35.050583 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050521 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:36.050501186 +0000 UTC m=+2.865695590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : configmap references non-existent config key: service-ca.crt Oct 30 12:51:35.050765 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050591 5049 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:35.050765 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050636 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls podName:fcd977c1-bd5b-466b-a9bd-aa4889ed86b4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:36.050618003 +0000 UTC m=+2.865812404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlskm" (UID: "fcd977c1-bd5b-466b-a9bd-aa4889ed86b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:35.050765 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050648 5049 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:51:35.050765 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050663 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:36.050652579 +0000 UTC m=+2.865847016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:35.050765 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050681 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls podName:116675da-08c0-4101-835c-48193661ccbf nodeName:}" failed. No retries permitted until 2026-10-30 12:51:36.050671408 +0000 UTC m=+2.865865810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls") pod "dns-default-94rws" (UID: "116675da-08c0-4101-835c-48193661ccbf") : secret "dns-default-metrics-tls" not found Oct 30 12:51:35.050765 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:35.050697 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls podName:482cc1b2-387f-4b1e-98f5-adada095ffec nodeName:}" failed. No retries permitted until 2026-10-30 12:51:36.05068832 +0000 UTC m=+2.865882723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jsqvh" (UID: "482cc1b2-387f-4b1e-98f5-adada095ffec") : secret "samples-operator-tls" not found Oct 30 12:51:35.058840 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.058818 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l7pxj"] Oct 30 12:51:35.063522 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:35.063500 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ad6fb3_8438_49d4_a7a8_cdac8c6c3bc3.slice/crio-5bd9342dabecfc2504a7ee538fd5a72178c7190b294c42f57fa1f3a510b99067 WatchSource:0}: Error finding container 5bd9342dabecfc2504a7ee538fd5a72178c7190b294c42f57fa1f3a510b99067: Status 404 returned error can't find the container with id 5bd9342dabecfc2504a7ee538fd5a72178c7190b294c42f57fa1f3a510b99067 Oct 30 12:51:35.063798 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.063778 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:35.392301 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.392219 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" event={"ID":"332bd965-1f4d-4749-8922-946a21e7ce75","Type":"ContainerStarted","Data":"5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88"} Oct 30 12:51:35.392301 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.392272 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" event={"ID":"332bd965-1f4d-4749-8922-946a21e7ce75","Type":"ContainerStarted","Data":"aee9e491c329d03d8788bc9356fd1b51d5076ca4ade3155990f1de2ee37d4262"} Oct 30 12:51:35.393622 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.393585 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" event={"ID":"caed0908-537e-4064-858e-f3f8a82b87b1","Type":"ContainerStarted","Data":"478b10c07789e1aef7f0549198e7b224788e45dd07a3d247b008f953bec7907f"} Oct 30 12:51:35.395109 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.395083 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7pxj" event={"ID":"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3","Type":"ContainerStarted","Data":"5bd9342dabecfc2504a7ee538fd5a72178c7190b294c42f57fa1f3a510b99067"} Oct 30 12:51:35.396787 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.396761 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" event={"ID":"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88","Type":"ContainerStarted","Data":"a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa"} Oct 30 12:51:35.396893 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.396809 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" event={"ID":"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88","Type":"ContainerStarted","Data":"ee00a8471e20afcee9cc6f824850213d5bc33bdb36fd9da2ebfcb5261b6fd56e"} Oct 30 12:51:35.399628 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.399611 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 12:51:35.399872 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.399845 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" event={"ID":"264f3a6d-7988-4ee8-bf15-bf13e991a63c","Type":"ContainerStarted","Data":"f9cb3540e243c95acd835277020ab19fbfb86eab17ef757e13e476d3b7a6bd0c"} Oct 30 12:51:35.428446 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.428410 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:35.612124 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.612065 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:35.774672 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.774590 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:35.928617 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:35.928585 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:36.062090 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.062057 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:36.062223 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.062102 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:36.062223 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.062135 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:36.062223 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.062161 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:36.062223 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.062195 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:36.062475 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062237 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:38.062214152 +0000 UTC m=+4.877408567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : configmap references non-existent config key: service-ca.crt Oct 30 12:51:36.062475 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.062287 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:36.062475 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062295 5049 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:36.062475 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062351 5049 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 12:51:36.062475 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062299 5049 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:36.062475 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062390 5049 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:36.062475 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062312 5049 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 12:51:36.062475 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062388 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls podName:fcd977c1-bd5b-466b-a9bd-aa4889ed86b4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:38.062374401 +0000 UTC m=+4.877568815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlskm" (UID: "fcd977c1-bd5b-466b-a9bd-aa4889ed86b4") : secret "cluster-monitoring-operator-tls" not found Oct 30 12:51:36.062475 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062478 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs podName:173ce15b-35dd-4a98-8f29-d13c62b56206 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:38.062464882 +0000 UTC m=+4.877659283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs") pod "router-default-df858cf77-mnl8f" (UID: "173ce15b-35dd-4a98-8f29-d13c62b56206") : secret "router-metrics-certs-default" not found Oct 30 12:51:36.062809 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062494 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls podName:116675da-08c0-4101-835c-48193661ccbf nodeName:}" failed. No retries permitted until 2026-10-30 12:51:38.062485699 +0000 UTC m=+4.877680103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls") pod "dns-default-94rws" (UID: "116675da-08c0-4101-835c-48193661ccbf") : secret "dns-default-metrics-tls" not found Oct 30 12:51:36.062809 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062508 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:38.062500296 +0000 UTC m=+4.877694750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:36.062809 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:36.062524 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls podName:482cc1b2-387f-4b1e-98f5-adada095ffec nodeName:}" failed. No retries permitted until 2026-10-30 12:51:38.062516481 +0000 UTC m=+4.877710882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jsqvh" (UID: "482cc1b2-387f-4b1e-98f5-adada095ffec") : secret "samples-operator-tls" not found Oct 30 12:51:36.140779 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.140752 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:36.312383 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.312354 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:36.407607 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.407566 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-f4hf7" event={"ID":"caed0908-537e-4064-858e-f3f8a82b87b1","Type":"ContainerStarted","Data":"cf836ad410101c0fd0a998dbe15b6194e3e4a185bb818049178b5645411ba8e6"} Oct 30 12:51:36.409387 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.409359 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7pxj" event={"ID":"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3","Type":"ContainerStarted","Data":"30482eea6be7ad37b86c89310fb786282436fa7b24afc680bdc410a8ff7fc862"} Oct 30 12:51:36.409480 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.409404 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7pxj" event={"ID":"b5ad6fb3-8438-49d4-a7a8-cdac8c6c3bc3","Type":"ContainerStarted","Data":"039a24705af03a1f85a30d931ebb5cb87f6b1482df0f7158771f64eb700cb580"} Oct 30 12:51:36.409480 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.409414 5049 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:51:36.409480 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.409370 5049 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:51:36.524466 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.524433 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:36.924949 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:36.924913 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:37.314217 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:37.314191 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:37.563489 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:37.563457 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:37.761840 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:37.761810 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:38.083581 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.083539 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:38.083765 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.083666 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:38.083765 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.083698 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:38.083765 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:38.083702 5049 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:38.083765 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.083731 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:38.083765 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.083759 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:38.084091 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:38.083783 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls podName:58b3cad0-f9c1-4547-bb19-f1fd20041eca nodeName:}" failed. No retries permitted until 2026-10-30 12:51:42.083760696 +0000 UTC m=+8.898955113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzlvd" (UID: "58b3cad0-f9c1-4547-bb19-f1fd20041eca") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:38.084091 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:38.083836 5049 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:38.084091 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.083837 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:38.084091 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:38.083899 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls podName:116675da-08c0-4101-835c-48193661ccbf nodeName:}" failed. No retries permitted until 2026-10-30 12:51:42.083884698 +0000 UTC m=+8.899079114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls") pod "dns-default-94rws" (UID: "116675da-08c0-4101-835c-48193661ccbf") : secret "dns-default-metrics-tls" not found Oct 30 12:51:38.084493 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.084471 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173ce15b-35dd-4a98-8f29-d13c62b56206-service-ca-bundle\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:38.086640 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.086620 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/482cc1b2-387f-4b1e-98f5-adada095ffec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jsqvh\" (UID: \"482cc1b2-387f-4b1e-98f5-adada095ffec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:38.086750 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.086695 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173ce15b-35dd-4a98-8f29-d13c62b56206-metrics-certs\") pod \"router-default-df858cf77-mnl8f\" (UID: \"173ce15b-35dd-4a98-8f29-d13c62b56206\") " pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:38.086813 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.086788 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcd977c1-bd5b-466b-a9bd-aa4889ed86b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlskm\" (UID: \"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:38.143988 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.143955 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:38.217145 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.217113 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:38.218148 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.218130 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" Oct 30 12:51:38.219795 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.219766 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" Oct 30 12:51:38.374834 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.374760 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:38.418534 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.418505 5049 generic.go:356] "Generic (PLEG): container finished" podID="fc1fa5e9-0704-4469-83b4-bdb65f799855" containerID="09f4d59355af2847938c07260800ecdcaf5161be90c95419c2f8e80a39f0abe9" exitCode=0 Oct 30 12:51:38.418667 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.418574 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6qdnl" event={"ID":"fc1fa5e9-0704-4469-83b4-bdb65f799855","Type":"ContainerDied","Data":"09f4d59355af2847938c07260800ecdcaf5161be90c95419c2f8e80a39f0abe9"} Oct 30 12:51:38.418919 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.418905 5049 scope.go:122] "RemoveContainer" containerID="09f4d59355af2847938c07260800ecdcaf5161be90c95419c2f8e80a39f0abe9" Oct 30 12:51:38.456396 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.456376 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:38.456549 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.456538 5049 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:51:38.535377 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.535343 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm"] Oct 30 12:51:38.545010 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.544976 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh"] Oct 30 12:51:38.555564 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.555541 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-df858cf77-mnl8f"] Oct 30 12:51:38.561831 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:38.561807 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd977c1_bd5b_466b_a9bd_aa4889ed86b4.slice/crio-1ba8393a3fbadc231329c350c142342f5b5aaab3f72ce0e99a07b5ad0dd4537a WatchSource:0}: Error finding container 1ba8393a3fbadc231329c350c142342f5b5aaab3f72ce0e99a07b5ad0dd4537a: Status 404 returned error can't find the container with id 1ba8393a3fbadc231329c350c142342f5b5aaab3f72ce0e99a07b5ad0dd4537a Oct 30 12:51:38.562089 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:38.562064 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173ce15b_35dd_4a98_8f29_d13c62b56206.slice/crio-490d80b2444b5e4ed190d1e72d0c8c94da08b5975413ac04be01ea92caa9bc83 WatchSource:0}: Error finding container 490d80b2444b5e4ed190d1e72d0c8c94da08b5975413ac04be01ea92caa9bc83: Status 404 returned error can't find the container with id 490d80b2444b5e4ed190d1e72d0c8c94da08b5975413ac04be01ea92caa9bc83 Oct 30 12:51:38.923347 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:38.923249 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:39.307011 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:39.306975 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:39.424223 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:39.424183 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" event={"ID":"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4","Type":"ContainerStarted","Data":"1ba8393a3fbadc231329c350c142342f5b5aaab3f72ce0e99a07b5ad0dd4537a"} Oct 30 12:51:39.426004 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:39.425974 5049 generic.go:356] "Generic (PLEG): container finished" podID="56a8078f-bd8e-42d8-86ef-4e0030e0ea76" containerID="4954ad3fecc2927d789fb6c5cb56c23d831c868a7e622bd8cf29caf09ae1a000" exitCode=0 Oct 30 12:51:39.426152 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:39.426043 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" event={"ID":"56a8078f-bd8e-42d8-86ef-4e0030e0ea76","Type":"ContainerDied","Data":"4954ad3fecc2927d789fb6c5cb56c23d831c868a7e622bd8cf29caf09ae1a000"} Oct 30 12:51:39.426474 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:39.426423 5049 scope.go:122] "RemoveContainer" containerID="4954ad3fecc2927d789fb6c5cb56c23d831c868a7e622bd8cf29caf09ae1a000" Oct 30 12:51:39.429755 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:39.429733 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6qdnl" event={"ID":"fc1fa5e9-0704-4469-83b4-bdb65f799855","Type":"ContainerStarted","Data":"5585a45b58d10a0efe84519594d28b7595c74c28d9bd80b22b7d462c7e220979"} Oct 30 12:51:39.431418 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:39.431394 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" event={"ID":"482cc1b2-387f-4b1e-98f5-adada095ffec","Type":"ContainerStarted","Data":"6ae75c42f791476e074486ab8017849171139d88fb44c79bb1bc9039211953ee"} Oct 30 12:51:39.433760 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:39.433734 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-df858cf77-mnl8f" event={"ID":"173ce15b-35dd-4a98-8f29-d13c62b56206","Type":"ContainerStarted","Data":"d9862fff26b7893b8072ce2c64348e36bf24448edf295f6c686fce24f800aef4"} Oct 30 12:51:39.433870 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:39.433778 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-df858cf77-mnl8f" event={"ID":"173ce15b-35dd-4a98-8f29-d13c62b56206","Type":"ContainerStarted","Data":"490d80b2444b5e4ed190d1e72d0c8c94da08b5975413ac04be01ea92caa9bc83"} Oct 30 12:51:39.775713 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:39.775633 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:40.119037 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.118995 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:40.217282 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.217257 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:40.220663 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.220447 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:40.326370 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.326335 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:40.449344 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.449141 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" event={"ID":"482cc1b2-387f-4b1e-98f5-adada095ffec","Type":"ContainerStarted","Data":"e77ceba57628402ceb3292e76319cc4451d0a3f920f2b17cbe4ae15b1279e5fe"} Oct 30 12:51:40.449344 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.449234 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jsqvh" event={"ID":"482cc1b2-387f-4b1e-98f5-adada095ffec","Type":"ContainerStarted","Data":"a4a9fa5a80f18524cb0ec96a51c5ae8fce2f65596486fdbc861825e9bbc89f2f"} Oct 30 12:51:40.451232 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.451206 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pnqvv" event={"ID":"56a8078f-bd8e-42d8-86ef-4e0030e0ea76","Type":"ContainerStarted","Data":"82ed9f5eac09e17b40c4ab2a25a4d40fbbca68e6e05af886a55d1a3d55264dad"} Oct 30 12:51:40.451457 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.451438 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:40.452870 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.452853 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-df858cf77-mnl8f" Oct 30 12:51:40.541998 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.541941 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:40.726346 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:40.726259 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.131162 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:41.131122 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.299492 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:41.299459 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.456476 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:41.456391 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlskm" event={"ID":"fcd977c1-bd5b-466b-a9bd-aa4889ed86b4","Type":"ContainerStarted","Data":"2c091cfb0925e572cd2cdd71e5ba717c6ff1947ad91c3a3660367356531675a1"} Oct 30 12:51:41.482572 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:41.482535 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.636713 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:41.636674 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.904918 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:41.904888 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.981556 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:41.981524 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:41.981746 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:41.981672 5049 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:51:41.985350 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:41.985307 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2xkr" Oct 30 12:51:42.057889 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.057857 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:51:42.124699 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.124664 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:42.124868 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.124811 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:42.127173 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.127152 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58b3cad0-f9c1-4547-bb19-f1fd20041eca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzlvd\" (UID: \"58b3cad0-f9c1-4547-bb19-f1fd20041eca\") " pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:42.127262 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.127193 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/116675da-08c0-4101-835c-48193661ccbf-metrics-tls\") pod \"dns-default-94rws\" (UID: \"116675da-08c0-4101-835c-48193661ccbf\") " pod="openshift-dns/dns-default-94rws" Oct 30 12:51:42.130124 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.130106 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94rws" Oct 30 12:51:42.131193 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.131176 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzlvd" Oct 30 12:51:42.142126 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.142101 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:42.317944 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.317902 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:42.373706 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.373670 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:51:42.432434 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.432363 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94rws"] Oct 30 12:51:42.435614 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:42.435588 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116675da_08c0_4101_835c_48193661ccbf.slice/crio-c8c66d483ae12ec3d0fc487f9a1465066dd941cb297d5d106e4101ec0a8b60dc WatchSource:0}: Error finding container c8c66d483ae12ec3d0fc487f9a1465066dd941cb297d5d106e4101ec0a8b60dc: Status 404 returned error can't find the container with id c8c66d483ae12ec3d0fc487f9a1465066dd941cb297d5d106e4101ec0a8b60dc Oct 30 12:51:42.441534 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.441509 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzlvd"] Oct 30 12:51:42.445208 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:42.445185 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b3cad0_f9c1_4547_bb19_f1fd20041eca.slice/crio-0957aac6a54bb4c0ac212806479e43f2636201f674dad17c4e54e848e0527a8c WatchSource:0}: Error finding container 0957aac6a54bb4c0ac212806479e43f2636201f674dad17c4e54e848e0527a8c: Status 404 returned error can't find the container with id 0957aac6a54bb4c0ac212806479e43f2636201f674dad17c4e54e848e0527a8c Oct 30 12:51:42.462655 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.462627 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzlvd" event={"ID":"58b3cad0-f9c1-4547-bb19-f1fd20041eca","Type":"ContainerStarted","Data":"0957aac6a54bb4c0ac212806479e43f2636201f674dad17c4e54e848e0527a8c"} Oct 30 12:51:42.463799 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.463773 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94rws" event={"ID":"116675da-08c0-4101-835c-48193661ccbf","Type":"ContainerStarted","Data":"c8c66d483ae12ec3d0fc487f9a1465066dd941cb297d5d106e4101ec0a8b60dc"} Oct 30 12:51:42.706717 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.706625 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:42.918170 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:42.918128 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:43.280043 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:43.280012 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:51:43.470148 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:43.470059 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzlvd" event={"ID":"58b3cad0-f9c1-4547-bb19-f1fd20041eca","Type":"ContainerStarted","Data":"42e8edd7f08472e0dbce594350396c54a7d33df7550cd2f37267b601b9e3c70f"} Oct 30 12:51:43.470148 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:43.470128 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzlvd" event={"ID":"58b3cad0-f9c1-4547-bb19-f1fd20041eca","Type":"ContainerStarted","Data":"16efd8a3628ed52d8fdff4fc43217d41d10d9b1714e5ca883ea6a3b50d4cface"} Oct 30 12:51:43.501201 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:43.501165 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:44.476385 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:44.476343 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94rws" event={"ID":"116675da-08c0-4101-835c-48193661ccbf","Type":"ContainerStarted","Data":"99c0f98a933e0870bcc1a76fc571625f26e2a60d06c375f43fca12d5567603d2"} Oct 30 12:51:44.476825 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:44.476409 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94rws" event={"ID":"116675da-08c0-4101-835c-48193661ccbf","Type":"ContainerStarted","Data":"725ed6e7861c3ecd324a0cc76cdc1fcc40b09b310fdf984904ff24214b1bcdab"} Oct 30 12:51:44.476825 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:44.476463 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-94rws" Oct 30 12:51:45.482447 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:45.482413 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzlvd" event={"ID":"58b3cad0-f9c1-4547-bb19-f1fd20041eca","Type":"ContainerStarted","Data":"e5647d820425210575bfdf032f46d2276b4ba0ae0903abfbdc806c5d30aa971d"} Oct 30 12:51:50.211963 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:50.211929 5049 patch_prober.go:29] interesting pod/image-registry-87dcb74f4-755j7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 12:51:50.212532 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:50.211985 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" podUID="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:51:50.641201 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:50.641166 5049 patch_prober.go:29] interesting pod/image-registry-85cb969f47-rcssw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 12:51:50.641390 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:50.641212 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" podUID="332bd965-1f4d-4749-8922-946a21e7ce75" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:51:52.062998 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:52.062961 5049 patch_prober.go:29] interesting pod/image-registry-87dcb74f4-755j7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 12:51:52.063443 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:52.063008 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" podUID="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:51:52.299597 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:52.299562 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cdptr" Oct 30 12:51:52.378255 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:52.378180 5049 patch_prober.go:29] interesting pod/image-registry-85cb969f47-rcssw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 12:51:52.378255 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:52.378229 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" podUID="332bd965-1f4d-4749-8922-946a21e7ce75" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:51:54.080534 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.080499 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-qd8mm-4xxx5"] Oct 30 12:51:54.091203 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.091176 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.113357 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.113246 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qd8mm-4xxx5"] Oct 30 12:51:54.114022 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.113996 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:54.132875 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.132840 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-87dcb74f4-755j7"] Oct 30 12:51:54.136244 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.133893 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-service-ca-bundle\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.136244 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.134014 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z52d6\" (UniqueName: \"kubernetes.io/projected/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-kube-api-access-z52d6\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.136244 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.134065 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-archives-path\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.136244 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.134088 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-trusted-ca-bundle\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.137541 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.137511 5049 patch_prober.go:29] interesting pod/image-registry-87dcb74f4-755j7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 12:51:54.137636 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.137556 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" podUID="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:51:54.235017 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.234978 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z52d6\" (UniqueName: \"kubernetes.io/projected/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-kube-api-access-z52d6\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.235212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.235038 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-archives-path\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.235212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.235054 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-trusted-ca-bundle\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.235212 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.235096 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-service-ca-bundle\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.235516 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.235492 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-archives-path\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.235936 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.235914 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-service-ca-bundle\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.236172 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.236146 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-trusted-ca-bundle\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.266841 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.266809 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z52d6\" (UniqueName: \"kubernetes.io/projected/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-kube-api-access-z52d6\") pod \"periodic-gathering-qd8mm-4xxx5\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.325291 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.325262 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:54.406021 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.405940 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:51:54.484193 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.484164 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-94rws" Oct 30 12:51:54.521108 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.521064 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:54.680964 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.680886 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:54.701205 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:54.701174 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qd8mm-4xxx5"] Oct 30 12:51:54.704593 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:51:54.704563 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e9f9d4_3b1f_4b11_abd0_3c5aa350c97c.slice/crio-900c8a2a6a153b167a4087dbd379bee4d86e62827e4985fa806b13320150cf72 WatchSource:0}: Error finding container 900c8a2a6a153b167a4087dbd379bee4d86e62827e4985fa806b13320150cf72: Status 404 returned error can't find the container with id 900c8a2a6a153b167a4087dbd379bee4d86e62827e4985fa806b13320150cf72 Oct 30 12:51:55.526464 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:55.526420 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" event={"ID":"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c","Type":"ContainerStarted","Data":"3f6ee40d4ced6002327023cdfe5d71e4af4bd36d8a8bfcfbf912adf6cec90480"} Oct 30 12:51:55.526859 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:55.526481 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" event={"ID":"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c","Type":"ContainerStarted","Data":"900c8a2a6a153b167a4087dbd379bee4d86e62827e4985fa806b13320150cf72"} Oct 30 12:51:55.563153 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:55.563096 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" podStartSLOduration=1.5630820189999999 podStartE2EDuration="1.563082019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:55.562081095 +0000 UTC m=+22.377275515" watchObservedRunningTime="2026-10-30 12:51:55.563082019 +0000 UTC m=+22.378276441" Oct 30 12:51:56.675459 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:51:56.675418 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:51:56.896597 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:56.896570 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp5jc" Oct 30 12:51:57.201927 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:57.201816 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94rws_116675da-08c0-4101-835c-48193661ccbf/dns/0.log" Oct 30 12:51:57.377103 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:57.377075 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94rws_116675da-08c0-4101-835c-48193661ccbf/kube-rbac-proxy/0.log" Oct 30 12:51:57.615961 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:57.615918 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:57.770798 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:57.770763 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:58.373658 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:58.373630 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v4l8n_d0471989-f151-47a5-9d39-97d85760da9e/dns-node-resolver/0.log" Oct 30 12:51:58.776747 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:58.776669 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85cb969f47-rcssw_332bd965-1f4d-4749-8922-946a21e7ce75/registry/0.log" Oct 30 12:51:58.881756 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:58.881722 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:58.973213 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:58.973180 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-87dcb74f4-755j7_1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/registry/0.log" Oct 30 12:51:59.034877 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:59.034806 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:59.172945 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:59.172919 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6p76v_126669c1-ee47-49ab-a6f3-3592a66b1fd2/node-ca/0.log" Oct 30 12:51:59.774073 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:51:59.774039 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-df858cf77-mnl8f_173ce15b-35dd-4a98-8f29-d13c62b56206/router/0.log" Oct 30 12:52:00.174105 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:00.174081 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pzn2g_0cbb909f-28ac-4b1e-9a95-5b8870886b61/serve-healthcheck-canary/0.log" Oct 30 12:52:00.850016 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:00.849986 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:02.089423 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.089382 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:02.279039 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.278958 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:02.378225 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.378196 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:52:02.379205 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.379185 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tb2dm"] Oct 30 12:52:02.412095 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.412059 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.456227 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.456198 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 12:52:02.456431 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.456302 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 12:52:02.456998 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.456977 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 12:52:02.457310 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.457290 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 12:52:02.457421 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.457351 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hxn87\"" Oct 30 12:52:02.494304 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.494270 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:02.502447 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.502421 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.502543 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.502480 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b483a39-dd4d-4049-ac91-8d566b1c01ab-metrics-client-ca\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.502543 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.502510 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b483a39-dd4d-4049-ac91-8d566b1c01ab-root\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.502629 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.502611 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-tls\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.502664 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.502641 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gk46\" (UniqueName: \"kubernetes.io/projected/2b483a39-dd4d-4049-ac91-8d566b1c01ab-kube-api-access-9gk46\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.502709 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.502665 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b483a39-dd4d-4049-ac91-8d566b1c01ab-sys\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.502819 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.502799 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-wtmp\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.502877 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.502833 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.502931 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.502916 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-textfile\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.603537 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603505 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.603719 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603550 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b483a39-dd4d-4049-ac91-8d566b1c01ab-metrics-client-ca\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.603719 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603567 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b483a39-dd4d-4049-ac91-8d566b1c01ab-root\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.603719 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603605 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-tls\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.603719 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603708 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b483a39-dd4d-4049-ac91-8d566b1c01ab-root\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.603896 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603711 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gk46\" (UniqueName: \"kubernetes.io/projected/2b483a39-dd4d-4049-ac91-8d566b1c01ab-kube-api-access-9gk46\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.603896 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603763 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b483a39-dd4d-4049-ac91-8d566b1c01ab-sys\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.603896 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603806 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-wtmp\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.603896 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603838 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.603896 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603867 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-textfile\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.604205 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603885 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b483a39-dd4d-4049-ac91-8d566b1c01ab-sys\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.604205 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.603996 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-wtmp\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.604205 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.604162 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-textfile\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.604353 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.604217 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b483a39-dd4d-4049-ac91-8d566b1c01ab-metrics-client-ca\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.604353 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.604217 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.605977 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.605955 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-tls\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.606070 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.606005 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b483a39-dd4d-4049-ac91-8d566b1c01ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.663982 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.663947 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gk46\" (UniqueName: \"kubernetes.io/projected/2b483a39-dd4d-4049-ac91-8d566b1c01ab-kube-api-access-9gk46\") pod \"node-exporter-tb2dm\" (UID: \"2b483a39-dd4d-4049-ac91-8d566b1c01ab\") " pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.758928 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.758894 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tb2dm" Oct 30 12:52:02.774572 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.774543 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:02.801415 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:52:02.801388 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b483a39_dd4d_4049_ac91_8d566b1c01ab.slice/crio-f0fc502bdfba4e1b44a6376122611b2044c12f3464b11322d274e90de37fddb8 WatchSource:0}: Error finding container f0fc502bdfba4e1b44a6376122611b2044c12f3464b11322d274e90de37fddb8: Status 404 returned error can't find the container with id f0fc502bdfba4e1b44a6376122611b2044c12f3464b11322d274e90de37fddb8 Oct 30 12:52:02.941410 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:02.941340 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:03.113333 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:03.113299 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:03.283754 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:03.283684 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b9v2t" Oct 30 12:52:03.333698 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:03.333629 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:03.506346 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:03.502905 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:03.560930 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:03.560809 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb2dm" event={"ID":"2b483a39-dd4d-4049-ac91-8d566b1c01ab","Type":"ContainerStarted","Data":"f0fc502bdfba4e1b44a6376122611b2044c12f3464b11322d274e90de37fddb8"} Oct 30 12:52:03.659944 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:03.659908 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:03.822838 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:03.822745 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.045824 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:04.045791 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.137776 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:04.137706 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:52:04.197051 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:04.197023 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.362771 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:04.362748 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.566427 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:04.566391 5049 generic.go:356] "Generic (PLEG): container finished" podID="2b483a39-dd4d-4049-ac91-8d566b1c01ab" containerID="a6a654b1236be72e40001a221d9133937bc440cebdf9e454d0bc2584e47ec771" exitCode=0 Oct 30 12:52:04.566599 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:04.566478 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb2dm" event={"ID":"2b483a39-dd4d-4049-ac91-8d566b1c01ab","Type":"ContainerDied","Data":"a6a654b1236be72e40001a221d9133937bc440cebdf9e454d0bc2584e47ec771"} Oct 30 12:52:05.516067 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:05.516037 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:05.574508 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:05.574474 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb2dm" event={"ID":"2b483a39-dd4d-4049-ac91-8d566b1c01ab","Type":"ContainerStarted","Data":"f92a8f210a90e44f0fb6bd79966bed91fce227965e3b6a956f1da310b5fb017e"} Oct 30 12:52:05.574701 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:05.574538 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb2dm" event={"ID":"2b483a39-dd4d-4049-ac91-8d566b1c01ab","Type":"ContainerStarted","Data":"7a5409084b687857b941dd033349915ecc90013dd737b352c3fd9bcfcefaa00d"} Oct 30 12:52:05.644530 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:05.644478 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tb2dm" podStartSLOduration=2.077548958 podStartE2EDuration="3.644462683s" totalImagesPullingTime="1.488754118s" totalInitContainerRuntime="78.159607ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:05.633518247 +0000 UTC m=+32.448712669" watchObservedRunningTime="2026-10-30 12:52:05.644462683 +0000 UTC m=+32.459657146" Oct 30 12:52:05.696831 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:05.696783 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:05.854981 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:05.854945 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:06.040371 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:06.040337 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:07.192370 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:07.192315 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:07.400575 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:07.400540 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:07.607531 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:07.607497 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:09.555997 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:09.555960 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:09.841569 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:09.841490 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:10.327611 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:10.327577 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:10.530229 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:10.530194 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:12.300001 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:12.299965 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:14.804262 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:14.804182 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:15.047890 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:15.047845 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:16.159602 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:16.159565 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85cb969f47-rcssw"] Oct 30 12:52:16.317213 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:16.317147 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:16.506670 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:16.506593 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:16.691644 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:16.691608 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:17.424188 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:17.424154 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:17.581693 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:17.581664 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:19.056278 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.056243 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:19.155308 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.155272 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" podUID="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" containerName="registry" containerID="cri-o://a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa" gracePeriod=30 Oct 30 12:52:19.234377 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.234347 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:19.396032 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.396006 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:52:19.462176 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.462146 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-trusted-ca\") pod \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " Oct 30 12:52:19.462370 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.462198 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-image-registry-private-configuration\") pod \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " Oct 30 12:52:19.462370 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.462225 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-installation-pull-secrets\") pod \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " Oct 30 12:52:19.462370 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.462260 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-ca-trust-extracted\") pod \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " Oct 30 12:52:19.462544 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.462399 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-bound-sa-token\") pod \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " Oct 30 12:52:19.462544 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.462429 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-certificates\") pod \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " Oct 30 12:52:19.462544 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.462457 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-kube-api-access-s6ml9\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-kube-api-access-s6ml9\") pod \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " Oct 30 12:52:19.462544 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.462497 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") pod \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\" (UID: \"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88\") " Oct 30 12:52:19.462730 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.462662 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-trusted-ca" pod "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:19.462811 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.462788 5049 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-trusted-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:19.463269 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.463241 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-certificates" pod "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:19.465553 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.465524 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls" pod "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:19.467402 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.467374 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-kube-api-access-s6ml9" pod "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88"). InnerVolumeSpecName "kube-api-access-s6ml9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:19.467402 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.467391 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-image-registry-private-configuration" pod "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:19.467562 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.467496 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-installation-pull-secrets" pod "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:19.468247 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.468227 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-bound-sa-token" pod "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:19.470126 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.470098 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-ca-trust-extracted" pod "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" (UID: "1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:19.563443 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.563405 5049 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-image-registry-private-configuration\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:19.563443 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.563443 5049 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-installation-pull-secrets\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:19.563619 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.563457 5049 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-ca-trust-extracted\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:19.563619 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.563470 5049 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-bound-sa-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:19.563619 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.563485 5049 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-certificates\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:19.563619 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.563499 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6ml9\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-kube-api-access-s6ml9\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:19.563619 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.563513 5049 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88-registry-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:19.633564 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.633480 5049 generic.go:356] "Generic (PLEG): container finished" podID="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" containerID="a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa" exitCode=0 Oct 30 12:52:19.633564 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.633529 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" Oct 30 12:52:19.633763 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.633563 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" event={"ID":"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88","Type":"ContainerDied","Data":"a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa"} Oct 30 12:52:19.633763 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.633617 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-87dcb74f4-755j7" event={"ID":"1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88","Type":"ContainerDied","Data":"ee00a8471e20afcee9cc6f824850213d5bc33bdb36fd9da2ebfcb5261b6fd56e"} Oct 30 12:52:19.633763 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.633639 5049 scope.go:122] "RemoveContainer" containerID="a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa" Oct 30 12:52:19.642181 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.642158 5049 scope.go:122] "RemoveContainer" containerID="a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa" Oct 30 12:52:19.642466 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:52:19.642448 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa\": container with ID starting with a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa not found: ID does not exist" containerID="a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa" Oct 30 12:52:19.642519 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.642474 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa"} err="failed to get container status \"a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa\": rpc error: code = NotFound desc = could not find container \"a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa\": container with ID starting with a29438198ca802131300bc6289836ba3f17ea2378d5effee14128153dbd8c9aa not found: ID does not exist" Oct 30 12:52:19.694746 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.694719 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-87dcb74f4-755j7"] Oct 30 12:52:19.697696 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.697675 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-87dcb74f4-755j7"] Oct 30 12:52:19.934078 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:19.933995 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:20.128995 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:20.128964 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:21.332669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:21.332635 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" path="/var/lib/kubelet/pods/1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88/volumes" Oct 30 12:52:22.989548 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:22.989518 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:23.161989 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:23.161958 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:25.698745 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:25.698712 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7mcfh-fnbft"] Oct 30 12:52:25.699232 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:25.698896 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-7mcfh-fnbft" podUID="4f13020b-562d-470d-9bc4-70a5dd26f1e9" containerName="insights-gathering" containerID="cri-o://6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2" gracePeriod=30 Oct 30 12:52:25.959512 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:25.959454 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:52:26.117868 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.117834 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-service-ca-bundle\") pod \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " Oct 30 12:52:26.118029 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.117880 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-trusted-ca-bundle\") pod \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " Oct 30 12:52:26.118029 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.117907 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f13020b-562d-470d-9bc4-70a5dd26f1e9-kube-api-access-4k2f6\" (UniqueName: \"kubernetes.io/projected/4f13020b-562d-470d-9bc4-70a5dd26f1e9-kube-api-access-4k2f6\") pod \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " Oct 30 12:52:26.118029 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.117959 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f13020b-562d-470d-9bc4-70a5dd26f1e9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f13020b-562d-470d-9bc4-70a5dd26f1e9-archives-path\") pod \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\" (UID: \"4f13020b-562d-470d-9bc4-70a5dd26f1e9\") " Oct 30 12:52:26.118272 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.118248 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-trusted-ca-bundle" pod "4f13020b-562d-470d-9bc4-70a5dd26f1e9" (UID: "4f13020b-562d-470d-9bc4-70a5dd26f1e9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:26.118393 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.118363 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-service-ca-bundle" pod "4f13020b-562d-470d-9bc4-70a5dd26f1e9" (UID: "4f13020b-562d-470d-9bc4-70a5dd26f1e9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:26.118510 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.118374 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f13020b-562d-470d-9bc4-70a5dd26f1e9-archives-path" pod "4f13020b-562d-470d-9bc4-70a5dd26f1e9" (UID: "4f13020b-562d-470d-9bc4-70a5dd26f1e9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:26.120175 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.120151 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f13020b-562d-470d-9bc4-70a5dd26f1e9-kube-api-access-4k2f6" pod "4f13020b-562d-470d-9bc4-70a5dd26f1e9" (UID: "4f13020b-562d-470d-9bc4-70a5dd26f1e9"). InnerVolumeSpecName "kube-api-access-4k2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:26.219369 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.219270 5049 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-service-ca-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:26.219369 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.219301 5049 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f13020b-562d-470d-9bc4-70a5dd26f1e9-trusted-ca-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:26.219369 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.219313 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k2f6\" (UniqueName: \"kubernetes.io/projected/4f13020b-562d-470d-9bc4-70a5dd26f1e9-kube-api-access-4k2f6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:26.219369 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.219341 5049 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f13020b-562d-470d-9bc4-70a5dd26f1e9-archives-path\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:26.664095 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.664060 5049 generic.go:356] "Generic (PLEG): container finished" podID="4f13020b-562d-470d-9bc4-70a5dd26f1e9" containerID="6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2" exitCode=2 Oct 30 12:52:26.664296 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.664115 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7mcfh-fnbft" Oct 30 12:52:26.664296 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.664142 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7mcfh-fnbft" event={"ID":"4f13020b-562d-470d-9bc4-70a5dd26f1e9","Type":"ContainerDied","Data":"6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2"} Oct 30 12:52:26.664296 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.664197 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7mcfh-fnbft" event={"ID":"4f13020b-562d-470d-9bc4-70a5dd26f1e9","Type":"ContainerDied","Data":"ef12761b54de310bb0455cd4b2c0354e74c304518a7bdb116398633ff40056ad"} Oct 30 12:52:26.664296 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.664217 5049 scope.go:122] "RemoveContainer" containerID="6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2" Oct 30 12:52:26.674776 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.674757 5049 scope.go:122] "RemoveContainer" containerID="6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2" Oct 30 12:52:26.675046 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:52:26.675026 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2\": container with ID starting with 6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2 not found: ID does not exist" containerID="6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2" Oct 30 12:52:26.675123 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.675063 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2"} err="failed to get container status \"6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2\": rpc error: code = NotFound desc = could not find container \"6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2\": container with ID starting with 6c8916fafcf1deef142d75a742720c53f4ddfa570043f97d78990c752654e1e2 not found: ID does not exist" Oct 30 12:52:26.677766 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:52:26.677742 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:52:26.709272 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.709240 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7mcfh-fnbft"] Oct 30 12:52:26.719279 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:26.719256 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-7mcfh-fnbft"] Oct 30 12:52:27.332249 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:27.332222 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f13020b-562d-470d-9bc4-70a5dd26f1e9" path="/var/lib/kubelet/pods/4f13020b-562d-470d-9bc4-70a5dd26f1e9/volumes" Oct 30 12:52:29.927939 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:29.927890 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:30.086544 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:30.086507 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:34.057753 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:34.057719 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:34.215827 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:34.215796 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:41.184842 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.184781 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" podUID="332bd965-1f4d-4749-8922-946a21e7ce75" containerName="registry" containerID="cri-o://5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88" gracePeriod=30 Oct 30 12:52:41.438933 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.438879 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:52:41.441861 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.441840 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-bound-sa-token\") pod \"332bd965-1f4d-4749-8922-946a21e7ce75\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " Oct 30 12:52:41.441982 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.441888 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-image-registry-private-configuration\") pod \"332bd965-1f4d-4749-8922-946a21e7ce75\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " Oct 30 12:52:41.441982 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.441925 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") pod \"332bd965-1f4d-4749-8922-946a21e7ce75\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " Oct 30 12:52:41.441982 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.441972 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-registry-certificates\") pod \"332bd965-1f4d-4749-8922-946a21e7ce75\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " Oct 30 12:52:41.442139 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.442010 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-kube-api-access-jqflv\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-kube-api-access-jqflv\") pod \"332bd965-1f4d-4749-8922-946a21e7ce75\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " Oct 30 12:52:41.442139 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.442049 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332bd965-1f4d-4749-8922-946a21e7ce75-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/332bd965-1f4d-4749-8922-946a21e7ce75-ca-trust-extracted\") pod \"332bd965-1f4d-4749-8922-946a21e7ce75\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " Oct 30 12:52:41.442139 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.442096 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-installation-pull-secrets\") pod \"332bd965-1f4d-4749-8922-946a21e7ce75\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " Oct 30 12:52:41.442296 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.442136 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-trusted-ca\") pod \"332bd965-1f4d-4749-8922-946a21e7ce75\" (UID: \"332bd965-1f4d-4749-8922-946a21e7ce75\") " Oct 30 12:52:41.442545 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.442511 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-registry-certificates" pod "332bd965-1f4d-4749-8922-946a21e7ce75" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:41.442777 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.442758 5049 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-registry-certificates\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:41.443179 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.443149 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-trusted-ca" pod "332bd965-1f4d-4749-8922-946a21e7ce75" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:41.444288 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.444255 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls" pod "332bd965-1f4d-4749-8922-946a21e7ce75" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:41.444646 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.444621 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-image-registry-private-configuration" pod "332bd965-1f4d-4749-8922-946a21e7ce75" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:41.444738 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.444660 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-bound-sa-token" pod "332bd965-1f4d-4749-8922-946a21e7ce75" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:41.444796 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.444765 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-kube-api-access-jqflv" pod "332bd965-1f4d-4749-8922-946a21e7ce75" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75"). InnerVolumeSpecName "kube-api-access-jqflv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:41.444796 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.444767 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-installation-pull-secrets" pod "332bd965-1f4d-4749-8922-946a21e7ce75" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:41.449361 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.449314 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332bd965-1f4d-4749-8922-946a21e7ce75-ca-trust-extracted" pod "332bd965-1f4d-4749-8922-946a21e7ce75" (UID: "332bd965-1f4d-4749-8922-946a21e7ce75"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:41.543728 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.543696 5049 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-bound-sa-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:41.543728 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.543728 5049 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-image-registry-private-configuration\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:41.543929 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.543739 5049 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-registry-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:41.543929 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.543749 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqflv\" (UniqueName: \"kubernetes.io/projected/332bd965-1f4d-4749-8922-946a21e7ce75-kube-api-access-jqflv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:41.543929 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.543758 5049 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/332bd965-1f4d-4749-8922-946a21e7ce75-ca-trust-extracted\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:41.543929 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.543769 5049 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/332bd965-1f4d-4749-8922-946a21e7ce75-installation-pull-secrets\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:41.543929 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.543779 5049 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/332bd965-1f4d-4749-8922-946a21e7ce75-trusted-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:52:41.728340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.728247 5049 generic.go:356] "Generic (PLEG): container finished" podID="332bd965-1f4d-4749-8922-946a21e7ce75" containerID="5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88" exitCode=0 Oct 30 12:52:41.728340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.728308 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" event={"ID":"332bd965-1f4d-4749-8922-946a21e7ce75","Type":"ContainerDied","Data":"5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88"} Oct 30 12:52:41.728507 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.728398 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" event={"ID":"332bd965-1f4d-4749-8922-946a21e7ce75","Type":"ContainerDied","Data":"aee9e491c329d03d8788bc9356fd1b51d5076ca4ade3155990f1de2ee37d4262"} Oct 30 12:52:41.728507 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.728308 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85cb969f47-rcssw" Oct 30 12:52:41.728507 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.728418 5049 scope.go:122] "RemoveContainer" containerID="5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88" Oct 30 12:52:41.736799 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.736780 5049 scope.go:122] "RemoveContainer" containerID="5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88" Oct 30 12:52:41.737059 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:52:41.737040 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88\": container with ID starting with 5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88 not found: ID does not exist" containerID="5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88" Oct 30 12:52:41.737110 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.737067 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88"} err="failed to get container status \"5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88\": rpc error: code = NotFound desc = could not find container \"5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88\": container with ID starting with 5b206e0ea3fd0404f6c65417a38f121f234cfeeba6f434a0b64593c3917e1f88 not found: ID does not exist" Oct 30 12:52:41.773627 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.773595 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85cb969f47-rcssw"] Oct 30 12:52:41.783697 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:41.783668 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85cb969f47-rcssw"] Oct 30 12:52:43.333154 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:43.333120 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332bd965-1f4d-4749-8922-946a21e7ce75" path="/var/lib/kubelet/pods/332bd965-1f4d-4749-8922-946a21e7ce75/volumes" Oct 30 12:52:46.022376 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:46.022333 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:46.207500 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:46.207467 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:49.050134 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:49.050099 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:49.206805 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:49.206769 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:53.774615 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:53.774579 5049 generic.go:356] "Generic (PLEG): container finished" podID="6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe" containerID="e3495df0605543f40ae65970bc49fe5b9f7e2ecf598307287a7ff003b92f8dc5" exitCode=0 Oct 30 12:52:53.775021 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:53.774652 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" event={"ID":"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe","Type":"ContainerDied","Data":"e3495df0605543f40ae65970bc49fe5b9f7e2ecf598307287a7ff003b92f8dc5"} Oct 30 12:52:53.775021 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:53.774995 5049 scope.go:122] "RemoveContainer" containerID="e3495df0605543f40ae65970bc49fe5b9f7e2ecf598307287a7ff003b92f8dc5" Oct 30 12:52:54.779850 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:54.779818 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-45lpn" event={"ID":"6a06a4b4-ab53-4ba8-803c-fd2fe71bd9fe","Type":"ContainerStarted","Data":"b6c363c929cdb3258dc57f64936561ac12c5736f164dfe88a0d608229db936a5"} Oct 30 12:52:56.675735 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:52:56.675700 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:52:56.737752 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:56.737718 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:56.896626 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:56.896592 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:57.053549 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:52:57.053516 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:01.869197 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:01.869159 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:02.028016 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:02.027984 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:04.284680 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:04.284641 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:05.244376 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:05.244335 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:05.403188 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:05.403152 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:05.926020 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:05.925993 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:06.075705 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:06.075665 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:14.491250 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:14.491199 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:14.639772 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:14.639737 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:19.095240 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:19.095208 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:19.257341 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:19.257290 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:20.880449 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:20.880413 5049 generic.go:356] "Generic (PLEG): container finished" podID="d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c" containerID="3f6ee40d4ced6002327023cdfe5d71e4af4bd36d8a8bfcfbf912adf6cec90480" exitCode=0 Oct 30 12:53:20.880895 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:20.880474 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" event={"ID":"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c","Type":"ContainerDied","Data":"3f6ee40d4ced6002327023cdfe5d71e4af4bd36d8a8bfcfbf912adf6cec90480"} Oct 30 12:53:22.015997 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.015973 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:53:22.181456 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.181376 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-kube-api-access-z52d6\" (UniqueName: \"kubernetes.io/projected/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-kube-api-access-z52d6\") pod \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " Oct 30 12:53:22.181456 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.181432 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-archives-path\") pod \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " Oct 30 12:53:22.181696 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.181524 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-service-ca-bundle\") pod \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " Oct 30 12:53:22.181696 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.181576 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-trusted-ca-bundle\") pod \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\" (UID: \"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c\") " Oct 30 12:53:22.181811 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.181790 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-archives-path" pod "d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c" (UID: "d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:22.181907 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.181888 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-service-ca-bundle" pod "d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c" (UID: "d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:22.182069 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.182020 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-trusted-ca-bundle" pod "d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c" (UID: "d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:22.183437 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.183416 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-kube-api-access-z52d6" pod "d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c" (UID: "d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c"). InnerVolumeSpecName "kube-api-access-z52d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:22.282343 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.282284 5049 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-trusted-ca-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:53:22.282343 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.282341 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z52d6\" (UniqueName: \"kubernetes.io/projected/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-kube-api-access-z52d6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:53:22.282547 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.282356 5049 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-archives-path\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:53:22.282547 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.282368 5049 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c-service-ca-bundle\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:53:22.480553 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.480475 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:22.890331 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.890285 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" event={"ID":"d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c","Type":"ContainerDied","Data":"900c8a2a6a153b167a4087dbd379bee4d86e62827e4985fa806b13320150cf72"} Oct 30 12:53:22.890486 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.890347 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="900c8a2a6a153b167a4087dbd379bee4d86e62827e4985fa806b13320150cf72" Oct 30 12:53:22.890486 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:22.890353 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qd8mm-4xxx5" Oct 30 12:53:23.120331 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:23.120285 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:23.501773 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:23.501740 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:23.657130 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:23.657095 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:26.675788 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:53:26.675753 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:53:39.138867 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:39.138831 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:39.294580 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:39.294545 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:45.736376 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:45.736294 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:45.899777 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:53:45.899744 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:56.676886 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:53:56.676850 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:54:23.276587 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:54:23.276545 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:26.675502 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:54:26.675464 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:54:52.294142 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:54:52.294089 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:52.455411 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:54:52.455375 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:56.675217 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:54:56.675181 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:54:57.801347 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:54:57.801286 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:58.003747 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:54:58.003709 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:06.984023 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:06.983963 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:14.433493 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:14.433417 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:15.279543 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:15.279507 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:17.836063 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:17.836030 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:17.988189 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:17.988163 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:20.160727 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:20.160689 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:20.362229 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:20.362196 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:20.640538 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:20.640503 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:20.933681 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:20.933574 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:22.468478 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:22.468445 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:22.634185 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:22.634154 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:23.367170 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:23.367138 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:24.846572 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:24.846536 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:25.013726 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:25.013692 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:26.680681 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:55:26.680638 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:55:29.700056 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:29.699998 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:29.926606 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:29.926564 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:30.471610 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.471573 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:30.607332 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607281 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-vptnq"] Oct 30 12:55:30.607650 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607635 5049 cpu_manager.go:414] "containerMap: removing container" podUID="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" containerName="registry" Oct 30 12:55:30.607694 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607652 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" containerName="registry" Oct 30 12:55:30.607694 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607666 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4f13020b-562d-470d-9bc4-70a5dd26f1e9" containerName="insights-gathering" Oct 30 12:55:30.607694 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607674 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f13020b-562d-470d-9bc4-70a5dd26f1e9" containerName="insights-gathering" Oct 30 12:55:30.607694 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607685 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c" containerName="insights-gathering" Oct 30 12:55:30.607694 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607691 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c" containerName="insights-gathering" Oct 30 12:55:30.607851 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607704 5049 cpu_manager.go:414] "containerMap: removing container" podUID="332bd965-1f4d-4749-8922-946a21e7ce75" containerName="registry" Oct 30 12:55:30.607851 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607710 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332bd965-1f4d-4749-8922-946a21e7ce75" containerName="registry" Oct 30 12:55:30.607851 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607781 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e9f9d4-3b1f-4b11-abd0-3c5aa350c97c" containerName="insights-gathering" Oct 30 12:55:30.607851 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607794 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="332bd965-1f4d-4749-8922-946a21e7ce75" containerName="registry" Oct 30 12:55:30.607851 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607807 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d13dd2f-93e1-42f7-a07c-d7e0ebf83e88" containerName="registry" Oct 30 12:55:30.607851 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.607816 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f13020b-562d-470d-9bc4-70a5dd26f1e9" containerName="insights-gathering" Oct 30 12:55:30.610822 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.610803 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vptnq" Oct 30 12:55:30.618761 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.618742 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6016f52c-e1a9-4e64-b61a-be98b76f262c-bound-sa-token\") pod \"cert-manager-597b95cb84-vptnq\" (UID: \"6016f52c-e1a9-4e64-b61a-be98b76f262c\") " pod="cert-manager/cert-manager-597b95cb84-vptnq" Oct 30 12:55:30.618845 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.618787 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6smg\" (UniqueName: \"kubernetes.io/projected/6016f52c-e1a9-4e64-b61a-be98b76f262c-kube-api-access-v6smg\") pod \"cert-manager-597b95cb84-vptnq\" (UID: \"6016f52c-e1a9-4e64-b61a-be98b76f262c\") " pod="cert-manager/cert-manager-597b95cb84-vptnq" Oct 30 12:55:30.635428 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.635407 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vptnq"] Oct 30 12:55:30.644131 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.644108 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 12:55:30.644131 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.644128 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nrct8\"" Oct 30 12:55:30.644259 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.644197 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 12:55:30.719908 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.719874 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6smg\" (UniqueName: \"kubernetes.io/projected/6016f52c-e1a9-4e64-b61a-be98b76f262c-kube-api-access-v6smg\") pod \"cert-manager-597b95cb84-vptnq\" (UID: \"6016f52c-e1a9-4e64-b61a-be98b76f262c\") " pod="cert-manager/cert-manager-597b95cb84-vptnq" Oct 30 12:55:30.720280 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.719932 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6016f52c-e1a9-4e64-b61a-be98b76f262c-bound-sa-token\") pod \"cert-manager-597b95cb84-vptnq\" (UID: \"6016f52c-e1a9-4e64-b61a-be98b76f262c\") " pod="cert-manager/cert-manager-597b95cb84-vptnq" Oct 30 12:55:30.741579 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.741514 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6smg\" (UniqueName: \"kubernetes.io/projected/6016f52c-e1a9-4e64-b61a-be98b76f262c-kube-api-access-v6smg\") pod \"cert-manager-597b95cb84-vptnq\" (UID: \"6016f52c-e1a9-4e64-b61a-be98b76f262c\") " pod="cert-manager/cert-manager-597b95cb84-vptnq" Oct 30 12:55:30.741682 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.741597 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6016f52c-e1a9-4e64-b61a-be98b76f262c-bound-sa-token\") pod \"cert-manager-597b95cb84-vptnq\" (UID: \"6016f52c-e1a9-4e64-b61a-be98b76f262c\") " pod="cert-manager/cert-manager-597b95cb84-vptnq" Oct 30 12:55:30.937745 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:30.937714 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vptnq" Oct 30 12:55:31.231655 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:31.231623 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vptnq"] Oct 30 12:55:31.234675 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:55:31.234642 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6016f52c_e1a9_4e64_b61a_be98b76f262c.slice/crio-a55e50770346528b987be4761774b1bb891981c1b8429ae4a6861d9bc7cb0b71 WatchSource:0}: Error finding container a55e50770346528b987be4761774b1bb891981c1b8429ae4a6861d9bc7cb0b71: Status 404 returned error can't find the container with id a55e50770346528b987be4761774b1bb891981c1b8429ae4a6861d9bc7cb0b71 Oct 30 12:55:31.374293 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:31.374261 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vptnq" event={"ID":"6016f52c-e1a9-4e64-b61a-be98b76f262c","Type":"ContainerStarted","Data":"a55e50770346528b987be4761774b1bb891981c1b8429ae4a6861d9bc7cb0b71"} Oct 30 12:55:32.056365 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:32.056311 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:32.205793 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:32.205759 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:34.387995 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:34.387959 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vptnq" event={"ID":"6016f52c-e1a9-4e64-b61a-be98b76f262c","Type":"ContainerStarted","Data":"a809a5b7e486c4baa8a134d0431937b86bba8edcb17fe9236a2ed7bb8b594350"} Oct 30 12:55:34.431243 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:34.431188 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-vptnq" podStartSLOduration=1.642579312 podStartE2EDuration="4.431175218s" totalImagesPullingTime="2.788595906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:34.430592421 +0000 UTC m=+241.245786836" watchObservedRunningTime="2026-10-30 12:55:34.431175218 +0000 UTC m=+241.246369637" Oct 30 12:55:39.385792 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:39.385758 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:39.546741 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:39.546708 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:40.994430 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:40.994401 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:51.671064 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:51.671026 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:51.887726 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:51.887581 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:52.058818 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.058790 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:52.437618 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.437534 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:52.526525 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.526494 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q"] Oct 30 12:55:52.529846 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.529825 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.556635 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.556610 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 12:55:52.556635 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.556633 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lvqkv\"" Oct 30 12:55:52.557165 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.557152 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 12:55:52.561489 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.561473 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 12:55:52.566836 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.566821 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 12:55:52.566880 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.566850 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 12:55:52.583966 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.583942 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q"] Oct 30 12:55:52.598595 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.598571 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-webhook-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.598697 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.598600 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fbbv\" (UniqueName: \"kubernetes.io/projected/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-kube-api-access-2fbbv\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.598697 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.598642 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-manifests\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.598812 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.598769 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.598873 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.598848 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-metrics-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.598924 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.598885 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-tmp\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.680594 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.680561 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:52.699395 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.699313 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-manifests\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.699395 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.699362 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.699395 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.699389 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-metrics-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.699628 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.699409 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-tmp\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.699628 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.699433 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-webhook-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.699628 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.699496 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fbbv\" (UniqueName: \"kubernetes.io/projected/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-kube-api-access-2fbbv\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.699807 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.699786 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-manifests\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.701783 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.701756 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-tmp\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.701888 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.701848 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-metrics-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.702056 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.702031 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.702150 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.702064 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-webhook-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.736843 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.736818 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fbbv\" (UniqueName: \"kubernetes.io/projected/d4a5b501-d88a-4063-9009-e59d5d0d0fe9-kube-api-access-2fbbv\") pod \"opendatahub-operator-controller-manager-678d69fff5-s2d8q\" (UID: \"d4a5b501-d88a-4063-9009-e59d5d0d0fe9\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.844231 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.844198 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:52.861380 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:52.861349 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:53.012991 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:53.012902 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:53.185053 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:53.185012 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:53.213258 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:53.213232 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q"] Oct 30 12:55:53.213935 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:55:53.213910 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a5b501_d88a_4063_9009_e59d5d0d0fe9.slice/crio-68c9aea806f79a52d90e941d35dc25ac44f2988be5d955b6f43f9ae5f6df4b5a WatchSource:0}: Error finding container 68c9aea806f79a52d90e941d35dc25ac44f2988be5d955b6f43f9ae5f6df4b5a: Status 404 returned error can't find the container with id 68c9aea806f79a52d90e941d35dc25ac44f2988be5d955b6f43f9ae5f6df4b5a Oct 30 12:55:53.384400 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:53.384362 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:53.468078 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:53.468045 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" event={"ID":"d4a5b501-d88a-4063-9009-e59d5d0d0fe9","Type":"ContainerStarted","Data":"68c9aea806f79a52d90e941d35dc25ac44f2988be5d955b6f43f9ae5f6df4b5a"} Oct 30 12:55:55.556798 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:55.556758 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:56.484738 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:56.484701 5049 generic.go:356] "Generic (PLEG): container finished" podID="d4a5b501-d88a-4063-9009-e59d5d0d0fe9" containerID="e084f54bc5b7f7287222507642b6c9a2bedfcee7604dfaefa92886ee0f237100" exitCode=0 Oct 30 12:55:56.484905 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:56.484788 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" event={"ID":"d4a5b501-d88a-4063-9009-e59d5d0d0fe9","Type":"ContainerDied","Data":"e084f54bc5b7f7287222507642b6c9a2bedfcee7604dfaefa92886ee0f237100"} Oct 30 12:55:56.675773 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:55:56.675685 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:55:57.061074 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:57.061038 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:57.490289 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:57.490211 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" event={"ID":"d4a5b501-d88a-4063-9009-e59d5d0d0fe9","Type":"ContainerStarted","Data":"224ddd2466484d620f6f008009eb2fc3204f10de1145b54cc7f8734461e34417"} Oct 30 12:55:57.490503 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:57.490359 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:55:57.584011 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:55:57.583954 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" podStartSLOduration=2.192100386 podStartE2EDuration="5.583935456s" totalImagesPullingTime="2.97624609s" totalInitContainerRuntime="415.58898ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:57.573044274 +0000 UTC m=+264.388238720" watchObservedRunningTime="2026-10-30 12:55:57.583935456 +0000 UTC m=+264.399129880" Oct 30 12:56:05.645111 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:05.645065 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:05.816514 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:05.816484 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:08.497162 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:08.497125 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-s2d8q" Oct 30 12:56:08.540436 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:08.540401 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:08.701055 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:08.701020 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:11.872965 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:11.872934 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:12.032988 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:12.032954 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:12.449843 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:12.449802 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:12.619947 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:12.619914 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:14.330836 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.330802 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j"] Oct 30 12:56:14.334521 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.334499 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.348566 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.348542 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 12:56:14.349390 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.349371 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 12:56:14.358303 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.358285 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 12:56:14.358420 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.358300 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 12:56:14.358420 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.358312 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-glws9\"" Oct 30 12:56:14.364875 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.364851 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j"] Oct 30 12:56:14.404338 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.404289 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:14.485372 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.485341 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.485372 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.485375 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.485575 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.485439 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.485575 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.485511 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4kxn\" (UniqueName: \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-kube-api-access-t4kxn\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.485575 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.485562 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.485669 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.485583 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.485705 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.485680 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.578028 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.577997 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5"] Oct 30 12:56:14.581388 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.581314 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.586638 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.586613 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.586745 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.586655 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.586745 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.586693 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.586745 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.586724 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4kxn\" (UniqueName: \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-kube-api-access-t4kxn\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.586863 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.586795 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.586863 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.586830 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.586936 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.586866 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.587603 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.587574 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.589381 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.589357 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.589728 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.589705 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.589799 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.589710 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.592041 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.592017 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.596766 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.596747 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:56:14.596840 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.596768 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 12:56:14.597478 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.597463 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 12:56:14.600305 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.600288 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 12:56:14.600402 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.600340 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rwc5b\"" Oct 30 12:56:14.600402 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.600395 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 12:56:14.612932 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.612914 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5"] Oct 30 12:56:14.625202 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.625181 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4kxn\" (UniqueName: \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-kube-api-access-t4kxn\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.625804 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.625788 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wwm7j\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.646679 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.646655 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:14.647299 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.647272 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:14.687388 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.687358 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d720519e-e7cf-49cc-8577-b3adb613f7e7-manager-config\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.687556 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.687400 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5xz\" (UniqueName: \"kubernetes.io/projected/d720519e-e7cf-49cc-8577-b3adb613f7e7-kube-api-access-tn5xz\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.687556 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.687432 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d720519e-e7cf-49cc-8577-b3adb613f7e7-metrics-cert\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.687641 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.687602 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d720519e-e7cf-49cc-8577-b3adb613f7e7-cert\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.788666 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.788634 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d720519e-e7cf-49cc-8577-b3adb613f7e7-manager-config\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.788816 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.788674 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5xz\" (UniqueName: \"kubernetes.io/projected/d720519e-e7cf-49cc-8577-b3adb613f7e7-kube-api-access-tn5xz\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.788816 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.788698 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d720519e-e7cf-49cc-8577-b3adb613f7e7-metrics-cert\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.788901 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.788810 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d720519e-e7cf-49cc-8577-b3adb613f7e7-cert\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.789391 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.789365 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d720519e-e7cf-49cc-8577-b3adb613f7e7-manager-config\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.791098 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.791074 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d720519e-e7cf-49cc-8577-b3adb613f7e7-metrics-cert\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.791202 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.791117 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d720519e-e7cf-49cc-8577-b3adb613f7e7-cert\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.809688 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.809661 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5xz\" (UniqueName: \"kubernetes.io/projected/d720519e-e7cf-49cc-8577-b3adb613f7e7-kube-api-access-tn5xz\") pod \"lws-controller-manager-6546cc79dc-dc8n5\" (UID: \"d720519e-e7cf-49cc-8577-b3adb613f7e7\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.908274 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.908202 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:14.957542 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:14.957459 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j"] Oct 30 12:56:14.964510 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:56:14.962492 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cb45c4_b725_4a2c_81fb_d14450aafe7b.slice/crio-f0319a71ec0357c6e512beb0c1699edaf882f86616b7d033f5fc619a8a24be91 WatchSource:0}: Error finding container f0319a71ec0357c6e512beb0c1699edaf882f86616b7d033f5fc619a8a24be91: Status 404 returned error can't find the container with id f0319a71ec0357c6e512beb0c1699edaf882f86616b7d033f5fc619a8a24be91 Oct 30 12:56:15.204742 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:15.204668 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5"] Oct 30 12:56:15.207809 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:56:15.207777 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd720519e_e7cf_49cc_8577_b3adb613f7e7.slice/crio-2d38b560b8305caa89c61b18b14d6bf8f561df2bc94db74404ee630ba90c470e WatchSource:0}: Error finding container 2d38b560b8305caa89c61b18b14d6bf8f561df2bc94db74404ee630ba90c470e: Status 404 returned error can't find the container with id 2d38b560b8305caa89c61b18b14d6bf8f561df2bc94db74404ee630ba90c470e Oct 30 12:56:15.562538 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:15.562504 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" event={"ID":"d720519e-e7cf-49cc-8577-b3adb613f7e7","Type":"ContainerStarted","Data":"2d38b560b8305caa89c61b18b14d6bf8f561df2bc94db74404ee630ba90c470e"} Oct 30 12:56:15.563819 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:15.563781 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" event={"ID":"b8cb45c4-b725-4a2c-81fb-d14450aafe7b","Type":"ContainerStarted","Data":"f0319a71ec0357c6e512beb0c1699edaf882f86616b7d033f5fc619a8a24be91"} Oct 30 12:56:15.718625 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:15.718583 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:15.969634 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:15.969550 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:16.120725 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:16.120688 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:16.474363 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:16.474309 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:18.070941 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:18.070514 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:18.070941 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:18.070591 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:18.581966 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:18.581929 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" event={"ID":"d720519e-e7cf-49cc-8577-b3adb613f7e7","Type":"ContainerStarted","Data":"c240b5a29fb94b7e06c15d997a467661d2d9789bbb422bed5f628ee456a02845"} Oct 30 12:56:18.582156 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:18.582063 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:18.583776 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:18.583749 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" event={"ID":"b8cb45c4-b725-4a2c-81fb-d14450aafe7b","Type":"ContainerStarted","Data":"43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae"} Oct 30 12:56:18.583906 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:18.583846 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:18.619762 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:18.619711 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" podStartSLOduration=1.65485511 podStartE2EDuration="4.61969682s" totalImagesPullingTime="2.96484171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:18.618630325 +0000 UTC m=+285.433824772" watchObservedRunningTime="2026-10-30 12:56:18.61969682 +0000 UTC m=+285.434891243" Oct 30 12:56:18.660499 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:18.660441 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" podStartSLOduration=1.555649045 podStartE2EDuration="4.660424753s" totalImagesPullingTime="3.104775708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:18.658309859 +0000 UTC m=+285.473504283" watchObservedRunningTime="2026-10-30 12:56:18.660424753 +0000 UTC m=+285.475619176" Oct 30 12:56:18.780160 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:18.780122 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:18.968981 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:18.968896 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:19.592672 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:19.592632 5049 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-wwm7j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:56:19.593065 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:19.592698 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" podUID="b8cb45c4-b725-4a2c-81fb-d14450aafe7b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:56:19.928230 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:19.928157 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:20.078933 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:20.078901 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:20.368582 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:20.368553 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:22.591723 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:22.591686 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:56:23.506083 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:23.506047 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:26.675831 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:56:26.675798 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:56:28.658364 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:28.658329 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:28.957662 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:28.957586 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:29.593805 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:29.593776 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-dc8n5" Oct 30 12:56:33.346191 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:33.346165 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 12:56:33.346628 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:56:33.346222 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 12:56:56.675488 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:56:56.675449 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:57:15.278947 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:15.278910 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:15.630152 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:15.630117 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:15.884976 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:15.884907 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:16.299856 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:16.299820 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:18.256208 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:18.256177 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:18.413090 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:18.413055 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:18.568673 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:18.568638 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:18.757233 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:18.757199 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:21.475589 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.475556 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j"] Oct 30 12:57:21.476009 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.475765 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" podUID="b8cb45c4-b725-4a2c-81fb-d14450aafe7b" containerName="discovery" containerID="cri-o://43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae" gracePeriod=30 Oct 30 12:57:21.722965 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.722943 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:57:21.756873 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.756795 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-ca-configmap\") pod \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " Oct 30 12:57:21.756995 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.756888 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-kubeconfig\") pod \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " Oct 30 12:57:21.756995 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.756927 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-kube-api-access-t4kxn\" (UniqueName: \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-kube-api-access-t4kxn\") pod \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " Oct 30 12:57:21.756995 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.756966 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-token\" (UniqueName: \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-token\") pod \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " Oct 30 12:57:21.757789 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.756994 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-cacerts\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-cacerts\") pod \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " Oct 30 12:57:21.757789 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.757041 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-dns-cert\") pod \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " Oct 30 12:57:21.757789 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.757125 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-local-certs\") pod \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\" (UID: \"b8cb45c4-b725-4a2c-81fb-d14450aafe7b\") " Oct 30 12:57:21.757789 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.757265 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-ca-configmap" pod "b8cb45c4-b725-4a2c-81fb-d14450aafe7b" (UID: "b8cb45c4-b725-4a2c-81fb-d14450aafe7b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:57:21.757789 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.757410 5049 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-ca-configmap\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:57:21.759712 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.759682 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-local-certs" pod "b8cb45c4-b725-4a2c-81fb-d14450aafe7b" (UID: "b8cb45c4-b725-4a2c-81fb-d14450aafe7b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:57:21.759842 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.759776 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-token" pod "b8cb45c4-b725-4a2c-81fb-d14450aafe7b" (UID: "b8cb45c4-b725-4a2c-81fb-d14450aafe7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:57:21.759909 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.759879 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-kubeconfig" pod "b8cb45c4-b725-4a2c-81fb-d14450aafe7b" (UID: "b8cb45c4-b725-4a2c-81fb-d14450aafe7b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:57:21.760133 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.760113 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-kube-api-access-t4kxn" pod "b8cb45c4-b725-4a2c-81fb-d14450aafe7b" (UID: "b8cb45c4-b725-4a2c-81fb-d14450aafe7b"). InnerVolumeSpecName "kube-api-access-t4kxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:57:21.760650 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.760631 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-dns-cert" pod "b8cb45c4-b725-4a2c-81fb-d14450aafe7b" (UID: "b8cb45c4-b725-4a2c-81fb-d14450aafe7b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:57:21.761130 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.761107 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-cacerts" pod "b8cb45c4-b725-4a2c-81fb-d14450aafe7b" (UID: "b8cb45c4-b725-4a2c-81fb-d14450aafe7b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:57:21.857763 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.857728 5049 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-kubeconfig\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:57:21.857763 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.857758 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4kxn\" (UniqueName: \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-kube-api-access-t4kxn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:57:21.857763 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.857767 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:57:21.858027 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.857778 5049 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-cacerts\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:57:21.858027 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.857786 5049 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-istio-csr-dns-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:57:21.858027 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.857794 5049 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cb45c4-b725-4a2c-81fb-d14450aafe7b-local-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:57:21.863307 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.863279 5049 generic.go:356] "Generic (PLEG): container finished" podID="b8cb45c4-b725-4a2c-81fb-d14450aafe7b" containerID="43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae" exitCode=0 Oct 30 12:57:21.863459 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.863345 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" Oct 30 12:57:21.863459 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.863359 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" event={"ID":"b8cb45c4-b725-4a2c-81fb-d14450aafe7b","Type":"ContainerDied","Data":"43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae"} Oct 30 12:57:21.863459 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.863410 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j" event={"ID":"b8cb45c4-b725-4a2c-81fb-d14450aafe7b","Type":"ContainerDied","Data":"f0319a71ec0357c6e512beb0c1699edaf882f86616b7d033f5fc619a8a24be91"} Oct 30 12:57:21.863459 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.863431 5049 scope.go:122] "RemoveContainer" containerID="43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae" Oct 30 12:57:21.872332 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.872304 5049 scope.go:122] "RemoveContainer" containerID="43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae" Oct 30 12:57:21.872600 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:57:21.872579 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae\": container with ID starting with 43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae not found: ID does not exist" containerID="43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae" Oct 30 12:57:21.872667 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.872615 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae"} err="failed to get container status \"43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae\": rpc error: code = NotFound desc = could not find container \"43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae\": container with ID starting with 43831284892ddf7a0e44f91ca2845c2ebf73f38a469ecd89e2b7420704d9d5ae not found: ID does not exist" Oct 30 12:57:21.912769 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.912741 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j"] Oct 30 12:57:21.919869 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:21.919836 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wwm7j"] Oct 30 12:57:23.332596 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:23.332560 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cb45c4-b725-4a2c-81fb-d14450aafe7b" path="/var/lib/kubelet/pods/b8cb45c4-b725-4a2c-81fb-d14450aafe7b/volumes" Oct 30 12:57:23.622420 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:57:23.622314 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:26.675609 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:57:26.675570 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:57:56.675350 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:57:56.675292 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:58:05.327309 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:05.327275 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:05.529620 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:05.529583 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:17.486619 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:17.486547 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:19.525382 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.525348 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq"] Oct 30 12:58:19.525768 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.525672 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b8cb45c4-b725-4a2c-81fb-d14450aafe7b" containerName="discovery" Oct 30 12:58:19.525768 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.525682 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cb45c4-b725-4a2c-81fb-d14450aafe7b" containerName="discovery" Oct 30 12:58:19.525768 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.525753 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cb45c4-b725-4a2c-81fb-d14450aafe7b" containerName="discovery" Oct 30 12:58:19.529051 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.529026 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq" Oct 30 12:58:19.556415 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.556386 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 12:58:19.556415 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.556408 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 12:58:19.556415 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.556393 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-swf9j\"" Oct 30 12:58:19.556415 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.556419 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 12:58:19.557793 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.557771 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq"] Oct 30 12:58:19.628469 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.628439 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86x4f\" (UniqueName: \"kubernetes.io/projected/0d345919-e44b-4028-80ee-f748cd3a53de-kube-api-access-86x4f\") pod \"dns-operator-controller-manager-65bfd79598-nddfq\" (UID: \"0d345919-e44b-4028-80ee-f748cd3a53de\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq" Oct 30 12:58:19.729181 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.729147 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86x4f\" (UniqueName: \"kubernetes.io/projected/0d345919-e44b-4028-80ee-f748cd3a53de-kube-api-access-86x4f\") pod \"dns-operator-controller-manager-65bfd79598-nddfq\" (UID: \"0d345919-e44b-4028-80ee-f748cd3a53de\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq" Oct 30 12:58:19.758352 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.758303 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86x4f\" (UniqueName: \"kubernetes.io/projected/0d345919-e44b-4028-80ee-f748cd3a53de-kube-api-access-86x4f\") pod \"dns-operator-controller-manager-65bfd79598-nddfq\" (UID: \"0d345919-e44b-4028-80ee-f748cd3a53de\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq" Oct 30 12:58:19.841632 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.841607 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq" Oct 30 12:58:19.877001 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:19.876974 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:20.153219 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:20.153137 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq"] Oct 30 12:58:20.156502 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:58:20.156471 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d345919_e44b_4028_80ee_f748cd3a53de.slice/crio-e083b3b24c1c26ef2d2f23cc9cf62122ec06cce2e00257a39d36ffca523259bb WatchSource:0}: Error finding container e083b3b24c1c26ef2d2f23cc9cf62122ec06cce2e00257a39d36ffca523259bb: Status 404 returned error can't find the container with id e083b3b24c1c26ef2d2f23cc9cf62122ec06cce2e00257a39d36ffca523259bb Oct 30 12:58:20.158867 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:20.158847 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:58:20.258914 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:20.258884 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:21.104514 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:21.104469 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq" event={"ID":"0d345919-e44b-4028-80ee-f748cd3a53de","Type":"ContainerStarted","Data":"e083b3b24c1c26ef2d2f23cc9cf62122ec06cce2e00257a39d36ffca523259bb"} Oct 30 12:58:21.471043 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:21.470964 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:22.163460 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:22.163428 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-q6vtv"] Oct 30 12:58:22.167438 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:22.167401 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-q6vtv" Oct 30 12:58:22.187984 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:22.187959 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ctlk4\"" Oct 30 12:58:22.200823 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:22.200784 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-q6vtv"] Oct 30 12:58:22.250501 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:22.250471 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26z5\" (UniqueName: \"kubernetes.io/projected/89cc7744-e78f-4659-a285-5d71985c6c12-kube-api-access-t26z5\") pod \"authorino-operator-68d6dd76dc-q6vtv\" (UID: \"89cc7744-e78f-4659-a285-5d71985c6c12\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-q6vtv" Oct 30 12:58:22.351358 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:22.351298 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26z5\" (UniqueName: \"kubernetes.io/projected/89cc7744-e78f-4659-a285-5d71985c6c12-kube-api-access-t26z5\") pod \"authorino-operator-68d6dd76dc-q6vtv\" (UID: \"89cc7744-e78f-4659-a285-5d71985c6c12\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-q6vtv" Oct 30 12:58:22.395522 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:22.395488 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26z5\" (UniqueName: \"kubernetes.io/projected/89cc7744-e78f-4659-a285-5d71985c6c12-kube-api-access-t26z5\") pod \"authorino-operator-68d6dd76dc-q6vtv\" (UID: \"89cc7744-e78f-4659-a285-5d71985c6c12\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-q6vtv" Oct 30 12:58:22.482457 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:22.482433 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-q6vtv" Oct 30 12:58:22.539597 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:22.539565 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:22.803864 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:22.803822 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-q6vtv"] Oct 30 12:58:22.807138 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:58:22.807105 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cc7744_e78f_4659_a285_5d71985c6c12.slice/crio-e975974f40d1ae822ce8885e969bbd331da1b457271120dfb99b98d79b14eda0 WatchSource:0}: Error finding container e975974f40d1ae822ce8885e969bbd331da1b457271120dfb99b98d79b14eda0: Status 404 returned error can't find the container with id e975974f40d1ae822ce8885e969bbd331da1b457271120dfb99b98d79b14eda0 Oct 30 12:58:23.057261 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:23.057152 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:23.115400 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:23.115358 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-q6vtv" event={"ID":"89cc7744-e78f-4659-a285-5d71985c6c12","Type":"ContainerStarted","Data":"e975974f40d1ae822ce8885e969bbd331da1b457271120dfb99b98d79b14eda0"} Oct 30 12:58:23.116909 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:23.116883 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq" event={"ID":"0d345919-e44b-4028-80ee-f748cd3a53de","Type":"ContainerStarted","Data":"df2a7b3132947549bc5007bfd15caf6c0e75833f5808d9067518aa53afd043b3"} Oct 30 12:58:23.117144 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:23.117127 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq" Oct 30 12:58:23.158968 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:23.158920 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq" podStartSLOduration=1.884537338 podStartE2EDuration="4.15890821s" totalImagesPullingTime="2.274370872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:23.1577418 +0000 UTC m=+409.972936223" watchObservedRunningTime="2026-10-30 12:58:23.15890821 +0000 UTC m=+409.974102632" Oct 30 12:58:23.496529 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:23.496444 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:23.816827 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:23.816792 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:24.072057 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:24.071980 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:24.122477 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:24.122443 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-q6vtv" event={"ID":"89cc7744-e78f-4659-a285-5d71985c6c12","Type":"ContainerStarted","Data":"c311f55d744343bed0b8714a45c71b07cc68e500acc6ca54dcf9862292b8f7b2"} Oct 30 12:58:24.122642 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:24.122523 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-q6vtv" Oct 30 12:58:24.174401 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:24.174358 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-q6vtv" podStartSLOduration=1.055625426 podStartE2EDuration="2.174346544s" totalImagesPullingTime="1.118721118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:24.174247888 +0000 UTC m=+410.989442312" watchObservedRunningTime="2026-10-30 12:58:24.174346544 +0000 UTC m=+410.989540957" Oct 30 12:58:24.385882 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:24.385807 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:26.675950 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:26.675916 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:58:27.332983 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.332955 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65"] Oct 30 12:58:27.336316 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.336301 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:27.369788 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.369768 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x7hnm\"" Oct 30 12:58:27.371589 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.371569 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65"] Oct 30 12:58:27.385351 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.385301 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:27.392546 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.392519 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71781481-f507-4004-a957-6aba8515ccb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" (UID: \"71781481-f507-4004-a957-6aba8515ccb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:27.392639 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.392596 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-897np\" (UniqueName: \"kubernetes.io/projected/71781481-f507-4004-a957-6aba8515ccb0-kube-api-access-897np\") pod \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" (UID: \"71781481-f507-4004-a957-6aba8515ccb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:27.493204 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.493175 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-897np\" (UniqueName: \"kubernetes.io/projected/71781481-f507-4004-a957-6aba8515ccb0-kube-api-access-897np\") pod \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" (UID: \"71781481-f507-4004-a957-6aba8515ccb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:27.493396 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.493215 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71781481-f507-4004-a957-6aba8515ccb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" (UID: \"71781481-f507-4004-a957-6aba8515ccb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:27.493637 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.493618 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71781481-f507-4004-a957-6aba8515ccb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" (UID: \"71781481-f507-4004-a957-6aba8515ccb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:27.541879 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.541851 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-897np\" (UniqueName: \"kubernetes.io/projected/71781481-f507-4004-a957-6aba8515ccb0-kube-api-access-897np\") pod \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" (UID: \"71781481-f507-4004-a957-6aba8515ccb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:27.648608 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.648512 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:27.781673 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.781638 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:27.981630 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:27.981556 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65"] Oct 30 12:58:27.984650 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:58:27.984616 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71781481_f507_4004_a957_6aba8515ccb0.slice/crio-fad5e41e18a145ac360abe0a1e10e6b0b49b19560bcc2d6b3a9035161539f477 WatchSource:0}: Error finding container fad5e41e18a145ac360abe0a1e10e6b0b49b19560bcc2d6b3a9035161539f477: Status 404 returned error can't find the container with id fad5e41e18a145ac360abe0a1e10e6b0b49b19560bcc2d6b3a9035161539f477 Oct 30 12:58:28.141154 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:28.141119 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" event={"ID":"71781481-f507-4004-a957-6aba8515ccb0","Type":"ContainerStarted","Data":"fad5e41e18a145ac360abe0a1e10e6b0b49b19560bcc2d6b3a9035161539f477"} Oct 30 12:58:28.790983 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:28.790950 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:29.166043 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:29.166010 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf"] Oct 30 12:58:29.169919 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:29.169895 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:29.198863 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:29.197736 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gmtdc\"" Oct 30 12:58:29.200374 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:29.200346 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf"] Oct 30 12:58:29.209008 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:29.208984 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v89b\" (UniqueName: \"kubernetes.io/projected/e9d7a2d9-fe9f-4272-ae32-07ac0f646bde-kube-api-access-5v89b\") pod \"limitador-operator-controller-manager-856dd584c5-xmfbf\" (UID: \"e9d7a2d9-fe9f-4272-ae32-07ac0f646bde\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:29.310524 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:29.310477 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v89b\" (UniqueName: \"kubernetes.io/projected/e9d7a2d9-fe9f-4272-ae32-07ac0f646bde-kube-api-access-5v89b\") pod \"limitador-operator-controller-manager-856dd584c5-xmfbf\" (UID: \"e9d7a2d9-fe9f-4272-ae32-07ac0f646bde\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:29.335299 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:29.335265 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v89b\" (UniqueName: \"kubernetes.io/projected/e9d7a2d9-fe9f-4272-ae32-07ac0f646bde-kube-api-access-5v89b\") pod \"limitador-operator-controller-manager-856dd584c5-xmfbf\" (UID: \"e9d7a2d9-fe9f-4272-ae32-07ac0f646bde\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:29.487715 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:29.487625 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:29.803219 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:29.803197 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf"] Oct 30 12:58:29.806310 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:58:29.806281 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d7a2d9_fe9f_4272_ae32_07ac0f646bde.slice/crio-99c96d80dac0ec025325fa4445fe4cac58c3531549e876c9abcb4534f7cd6c10 WatchSource:0}: Error finding container 99c96d80dac0ec025325fa4445fe4cac58c3531549e876c9abcb4534f7cd6c10: Status 404 returned error can't find the container with id 99c96d80dac0ec025325fa4445fe4cac58c3531549e876c9abcb4534f7cd6c10 Oct 30 12:58:30.152340 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:30.152239 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" event={"ID":"e9d7a2d9-fe9f-4272-ae32-07ac0f646bde","Type":"ContainerStarted","Data":"99c96d80dac0ec025325fa4445fe4cac58c3531549e876c9abcb4534f7cd6c10"} Oct 30 12:58:30.292035 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:30.291961 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:30.579518 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:30.579481 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:31.159386 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:31.159344 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" event={"ID":"e9d7a2d9-fe9f-4272-ae32-07ac0f646bde","Type":"ContainerStarted","Data":"18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9"} Oct 30 12:58:31.159814 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:31.159542 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:31.200759 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:31.200704 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" podStartSLOduration=0.993653659 podStartE2EDuration="2.200688184s" totalImagesPullingTime="1.207034525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:31.198890741 +0000 UTC m=+418.014085165" watchObservedRunningTime="2026-10-30 12:58:31.200688184 +0000 UTC m=+418.015882608" Oct 30 12:58:32.165369 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:32.165302 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" event={"ID":"71781481-f507-4004-a957-6aba8515ccb0","Type":"ContainerStarted","Data":"c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5"} Oct 30 12:58:32.165951 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:32.165570 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:32.243243 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:32.243188 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" podStartSLOduration=1.746034956 podStartE2EDuration="5.24317635s" totalImagesPullingTime="3.497141394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:32.232950166 +0000 UTC m=+419.048144589" watchObservedRunningTime="2026-10-30 12:58:32.24317635 +0000 UTC m=+419.058370773" Oct 30 12:58:34.125357 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:34.125300 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nddfq" Oct 30 12:58:35.084735 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:35.084700 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:35.129441 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:35.129408 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-q6vtv" Oct 30 12:58:35.380992 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:35.380912 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:35.617141 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:35.617106 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:40.076206 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:40.076172 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:40.445855 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:40.445766 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:42.167845 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:42.167817 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:43.171802 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:43.171770 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:45.642439 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:45.642404 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98"] Oct 30 12:58:45.645973 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:45.645953 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:45.748840 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:45.748802 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98"] Oct 30 12:58:45.752495 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:45.752469 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-cpc98\" (UID: \"fa36ac34-1cd2-44fd-b4b7-f8483469b2c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:45.752617 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:45.752512 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6k2\" (UniqueName: \"kubernetes.io/projected/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-kube-api-access-bm6k2\") pod \"kuadrant-operator-controller-manager-858bb878d8-cpc98\" (UID: \"fa36ac34-1cd2-44fd-b4b7-f8483469b2c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:45.852993 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:45.852961 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-cpc98\" (UID: \"fa36ac34-1cd2-44fd-b4b7-f8483469b2c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:45.853157 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:45.853006 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6k2\" (UniqueName: \"kubernetes.io/projected/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-kube-api-access-bm6k2\") pod \"kuadrant-operator-controller-manager-858bb878d8-cpc98\" (UID: \"fa36ac34-1cd2-44fd-b4b7-f8483469b2c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:45.853381 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:45.853361 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-cpc98\" (UID: \"fa36ac34-1cd2-44fd-b4b7-f8483469b2c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:45.876762 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:45.876729 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6k2\" (UniqueName: \"kubernetes.io/projected/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-kube-api-access-bm6k2\") pod \"kuadrant-operator-controller-manager-858bb878d8-cpc98\" (UID: \"fa36ac34-1cd2-44fd-b4b7-f8483469b2c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:45.958711 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:45.958617 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:46.295965 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.295934 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98"] Oct 30 12:58:46.299603 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:58:46.299574 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa36ac34_1cd2_44fd_b4b7_f8483469b2c6.slice/crio-19138cdb8f3d78b39b420ac1ee59c655a5104ad5e61356d4b8240f41356db19e WatchSource:0}: Error finding container 19138cdb8f3d78b39b420ac1ee59c655a5104ad5e61356d4b8240f41356db19e: Status 404 returned error can't find the container with id 19138cdb8f3d78b39b420ac1ee59c655a5104ad5e61356d4b8240f41356db19e Oct 30 12:58:46.897776 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.897742 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65"] Oct 30 12:58:46.898298 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.897949 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" podUID="71781481-f507-4004-a957-6aba8515ccb0" containerName="manager" containerID="cri-o://c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5" gracePeriod=2 Oct 30 12:58:46.899087 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.899061 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65"] Oct 30 12:58:46.910164 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:46.909790 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="71781481-f507-4004-a957-6aba8515ccb0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:46.919309 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.919285 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98"] Oct 30 12:58:46.930992 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.930968 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl"] Oct 30 12:58:46.931380 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.931364 5049 cpu_manager.go:414] "containerMap: removing container" podUID="71781481-f507-4004-a957-6aba8515ccb0" containerName="manager" Oct 30 12:58:46.931468 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.931382 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71781481-f507-4004-a957-6aba8515ccb0" containerName="manager" Oct 30 12:58:46.931524 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.931477 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="71781481-f507-4004-a957-6aba8515ccb0" containerName="manager" Oct 30 12:58:46.934756 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.934728 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98"] Oct 30 12:58:46.934863 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.934838 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:46.946846 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:46.946818 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="71781481-f507-4004-a957-6aba8515ccb0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:46.961638 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.961601 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h"] Oct 30 12:58:46.962515 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.962493 5049 cpu_manager.go:414] "containerMap: removing container" podUID="fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" containerName="manager" Oct 30 12:58:46.962515 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.962514 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" containerName="manager" Oct 30 12:58:46.962671 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.962628 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" containerName="manager" Oct 30 12:58:46.966406 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.966380 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:58:46.966763 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.966741 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf"] Oct 30 12:58:46.966928 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.966900 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" podUID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" containerName="manager" containerID="cri-o://18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9" gracePeriod=2 Oct 30 12:58:46.977703 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.977417 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl"] Oct 30 12:58:46.977933 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.977910 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf"] Oct 30 12:58:46.996684 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:46.996661 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h"] Oct 30 12:58:47.011927 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.011900 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8"] Oct 30 12:58:47.012309 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.012288 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" containerName="manager" Oct 30 12:58:47.012309 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.012307 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" containerName="manager" Oct 30 12:58:47.012527 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.012437 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" containerName="manager" Oct 30 12:58:47.016103 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.016081 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8" Oct 30 12:58:47.023187 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.023154 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="71781481-f507-4004-a957-6aba8515ccb0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:47.055685 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.055647 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8"] Oct 30 12:58:47.062934 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.062909 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bhzc\" (UniqueName: \"kubernetes.io/projected/4519e0b9-67ff-4ac5-9da9-af1c10707a29-kube-api-access-2bhzc\") pod \"kuadrant-operator-controller-manager-858bb878d8-7gf8h\" (UID: \"4519e0b9-67ff-4ac5-9da9-af1c10707a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:58:47.063068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.062964 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1498d838-1fae-4b00-aaff-1adba5a4614f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-rd5sl\" (UID: \"1498d838-1fae-4b00-aaff-1adba5a4614f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:47.063068 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.063041 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh75l\" (UniqueName: \"kubernetes.io/projected/1498d838-1fae-4b00-aaff-1adba5a4614f-kube-api-access-vh75l\") pod \"kuadrant-operator-controller-manager-6b6fb56846-rd5sl\" (UID: \"1498d838-1fae-4b00-aaff-1adba5a4614f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:47.063190 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.063070 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4519e0b9-67ff-4ac5-9da9-af1c10707a29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-7gf8h\" (UID: \"4519e0b9-67ff-4ac5-9da9-af1c10707a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:58:47.094544 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.094511 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-xmfbf\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:47.106096 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.106065 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="71781481-f507-4004-a957-6aba8515ccb0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:47.164513 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.163956 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bhzc\" (UniqueName: \"kubernetes.io/projected/4519e0b9-67ff-4ac5-9da9-af1c10707a29-kube-api-access-2bhzc\") pod \"kuadrant-operator-controller-manager-858bb878d8-7gf8h\" (UID: \"4519e0b9-67ff-4ac5-9da9-af1c10707a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:58:47.164513 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.164040 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1498d838-1fae-4b00-aaff-1adba5a4614f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-rd5sl\" (UID: \"1498d838-1fae-4b00-aaff-1adba5a4614f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:47.164513 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.164069 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh75l\" (UniqueName: \"kubernetes.io/projected/1498d838-1fae-4b00-aaff-1adba5a4614f-kube-api-access-vh75l\") pod \"kuadrant-operator-controller-manager-6b6fb56846-rd5sl\" (UID: \"1498d838-1fae-4b00-aaff-1adba5a4614f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:47.164513 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.164095 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4519e0b9-67ff-4ac5-9da9-af1c10707a29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-7gf8h\" (UID: \"4519e0b9-67ff-4ac5-9da9-af1c10707a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:58:47.164513 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.164135 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9566\" (UniqueName: \"kubernetes.io/projected/ac4d89bd-42a7-4527-8838-5fdf95fd54ee-kube-api-access-c9566\") pod \"limitador-operator-controller-manager-856dd584c5-mv4d8\" (UID: \"ac4d89bd-42a7-4527-8838-5fdf95fd54ee\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8" Oct 30 12:58:47.164768 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.164623 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4519e0b9-67ff-4ac5-9da9-af1c10707a29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-858bb878d8-7gf8h\" (UID: \"4519e0b9-67ff-4ac5-9da9-af1c10707a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:58:47.164768 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.164674 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1498d838-1fae-4b00-aaff-1adba5a4614f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-rd5sl\" (UID: \"1498d838-1fae-4b00-aaff-1adba5a4614f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:47.173621 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.173602 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:47.192905 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.192870 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-xmfbf\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:47.201596 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.201572 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh75l\" (UniqueName: \"kubernetes.io/projected/1498d838-1fae-4b00-aaff-1adba5a4614f-kube-api-access-vh75l\") pod \"kuadrant-operator-controller-manager-6b6fb56846-rd5sl\" (UID: \"1498d838-1fae-4b00-aaff-1adba5a4614f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:47.203059 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.203038 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bhzc\" (UniqueName: \"kubernetes.io/projected/4519e0b9-67ff-4ac5-9da9-af1c10707a29-kube-api-access-2bhzc\") pod \"kuadrant-operator-controller-manager-858bb878d8-7gf8h\" (UID: \"4519e0b9-67ff-4ac5-9da9-af1c10707a29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:58:47.205288 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.205255 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-xmfbf\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:47.217063 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.217040 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="71781481-f507-4004-a957-6aba8515ccb0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:47.218337 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.218310 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:47.230506 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.230484 5049 generic.go:356] "Generic (PLEG): container finished" podID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" containerID="18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9" exitCode=0 Oct 30 12:58:47.230600 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.230526 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:47.230600 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.230583 5049 scope.go:122] "RemoveContainer" containerID="18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9" Oct 30 12:58:47.231895 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.231876 5049 generic.go:356] "Generic (PLEG): container finished" podID="71781481-f507-4004-a957-6aba8515ccb0" containerID="c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5" exitCode=0 Oct 30 12:58:47.231978 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.231916 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:47.232428 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.232399 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="71781481-f507-4004-a957-6aba8515ccb0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:47.238755 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.238728 5049 scope.go:122] "RemoveContainer" containerID="18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9" Oct 30 12:58:47.239025 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.239009 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9\": container with ID starting with 18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9 not found: ID does not exist" containerID="18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9" Oct 30 12:58:47.239062 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.239034 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9"} err="failed to get container status \"18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9\": rpc error: code = NotFound desc = could not find container \"18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9\": container with ID starting with 18b3de12fad7fba9a0e1793d4b744d5feeb75903eb3a26f7b4c616ec9c0162d9 not found: ID does not exist" Oct 30 12:58:47.239062 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.239050 5049 scope.go:122] "RemoveContainer" containerID="c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5" Oct 30 12:58:47.244149 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.244128 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-xmfbf\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:47.247916 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.247885 5049 scope.go:122] "RemoveContainer" containerID="c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5" Oct 30 12:58:47.248166 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.248147 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5\": container with ID starting with c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5 not found: ID does not exist" containerID="c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5" Oct 30 12:58:47.248211 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.248174 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5"} err="failed to get container status \"c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5\": rpc error: code = NotFound desc = could not find container \"c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5\": container with ID starting with c1cf4f6fce7ff427f3287a228e0d86c763fb33a2b56826abba3b0281f5b30fe5 not found: ID does not exist" Oct 30 12:58:47.259088 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.259066 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-xmfbf\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xmfbf" Oct 30 12:58:47.265400 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.265382 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71781481-f507-4004-a957-6aba8515ccb0-kube-api-access-897np\" (UniqueName: \"kubernetes.io/projected/71781481-f507-4004-a957-6aba8515ccb0-kube-api-access-897np\") pod \"71781481-f507-4004-a957-6aba8515ccb0\" (UID: \"71781481-f507-4004-a957-6aba8515ccb0\") " Oct 30 12:58:47.265470 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.265456 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71781481-f507-4004-a957-6aba8515ccb0-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71781481-f507-4004-a957-6aba8515ccb0-extensions-socket-volume\") pod \"71781481-f507-4004-a957-6aba8515ccb0\" (UID: \"71781481-f507-4004-a957-6aba8515ccb0\") " Oct 30 12:58:47.265661 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.265644 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9566\" (UniqueName: \"kubernetes.io/projected/ac4d89bd-42a7-4527-8838-5fdf95fd54ee-kube-api-access-c9566\") pod \"limitador-operator-controller-manager-856dd584c5-mv4d8\" (UID: \"ac4d89bd-42a7-4527-8838-5fdf95fd54ee\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8" Oct 30 12:58:47.265869 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.265846 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71781481-f507-4004-a957-6aba8515ccb0-extensions-socket-volume" pod "71781481-f507-4004-a957-6aba8515ccb0" (UID: "71781481-f507-4004-a957-6aba8515ccb0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:47.267363 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.267343 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71781481-f507-4004-a957-6aba8515ccb0-kube-api-access-897np" pod "71781481-f507-4004-a957-6aba8515ccb0" (UID: "71781481-f507-4004-a957-6aba8515ccb0"). InnerVolumeSpecName "kube-api-access-897np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:58:47.270905 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.270886 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-75t65\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="71781481-f507-4004-a957-6aba8515ccb0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-75t65" Oct 30 12:58:47.284418 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:47.284398 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-858bb878d8-cpc98\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:47.296800 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.296777 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9566\" (UniqueName: \"kubernetes.io/projected/ac4d89bd-42a7-4527-8838-5fdf95fd54ee-kube-api-access-c9566\") pod \"limitador-operator-controller-manager-856dd584c5-mv4d8\" (UID: \"ac4d89bd-42a7-4527-8838-5fdf95fd54ee\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8" Oct 30 12:58:47.333754 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.333729 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71781481-f507-4004-a957-6aba8515ccb0" path="/var/lib/kubelet/pods/71781481-f507-4004-a957-6aba8515ccb0/volumes" Oct 30 12:58:47.345581 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.345552 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:47.350857 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.350834 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:47.357854 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.357831 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:58:47.363930 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.363903 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8" Oct 30 12:58:47.366072 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.366049 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d7a2d9-fe9f-4272-ae32-07ac0f646bde-kube-api-access-5v89b\" (UniqueName: \"kubernetes.io/projected/e9d7a2d9-fe9f-4272-ae32-07ac0f646bde-kube-api-access-5v89b\") pod \"e9d7a2d9-fe9f-4272-ae32-07ac0f646bde\" (UID: \"e9d7a2d9-fe9f-4272-ae32-07ac0f646bde\") " Oct 30 12:58:47.366230 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.366214 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-897np\" (UniqueName: \"kubernetes.io/projected/71781481-f507-4004-a957-6aba8515ccb0-kube-api-access-897np\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:58:47.366230 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.366229 5049 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71781481-f507-4004-a957-6aba8515ccb0-extensions-socket-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:58:47.368581 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.368557 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d7a2d9-fe9f-4272-ae32-07ac0f646bde-kube-api-access-5v89b" pod "e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" (UID: "e9d7a2d9-fe9f-4272-ae32-07ac0f646bde"). InnerVolumeSpecName "kube-api-access-5v89b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:58:47.466972 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.466945 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v89b\" (UniqueName: \"kubernetes.io/projected/e9d7a2d9-fe9f-4272-ae32-07ac0f646bde-kube-api-access-5v89b\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:58:47.625755 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.625721 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:47.667484 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.667410 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl"] Oct 30 12:58:47.674393 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:58:47.674365 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1498d838_1fae_4b00_aaff_1adba5a4614f.slice/crio-abba3650193426407735088e383a83e427076223eb7a43d60e162feefb579da7 WatchSource:0}: Error finding container abba3650193426407735088e383a83e427076223eb7a43d60e162feefb579da7: Status 404 returned error can't find the container with id abba3650193426407735088e383a83e427076223eb7a43d60e162feefb579da7 Oct 30 12:58:47.687940 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.687918 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8"] Oct 30 12:58:47.690218 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:58:47.690192 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4d89bd_42a7_4527_8838_5fdf95fd54ee.slice/crio-df22a425b332f5cd8556e97c540b9aa7c706e3d9e2c25843c3456e0e2b11e5c2 WatchSource:0}: Error finding container df22a425b332f5cd8556e97c540b9aa7c706e3d9e2c25843c3456e0e2b11e5c2: Status 404 returned error can't find the container with id df22a425b332f5cd8556e97c540b9aa7c706e3d9e2c25843c3456e0e2b11e5c2 Oct 30 12:58:47.706236 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:47.706214 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h"] Oct 30 12:58:47.710400 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:58:47.710376 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4519e0b9_67ff_4ac5_9da9_af1c10707a29.slice/crio-a4474dfcc9e9b6909285900de3d54f14e0cf8c5d022527dc93eb36171e761fbc WatchSource:0}: Error finding container a4474dfcc9e9b6909285900de3d54f14e0cf8c5d022527dc93eb36171e761fbc: Status 404 returned error can't find the container with id a4474dfcc9e9b6909285900de3d54f14e0cf8c5d022527dc93eb36171e761fbc Oct 30 12:58:48.240077 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.240000 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8" event={"ID":"ac4d89bd-42a7-4527-8838-5fdf95fd54ee","Type":"ContainerStarted","Data":"9f4bd05633ed5e0453cae5880e1c67297cecf37e5bb1958074387ed10393418b"} Oct 30 12:58:48.240077 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.240049 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8" event={"ID":"ac4d89bd-42a7-4527-8838-5fdf95fd54ee","Type":"ContainerStarted","Data":"df22a425b332f5cd8556e97c540b9aa7c706e3d9e2c25843c3456e0e2b11e5c2"} Oct 30 12:58:48.240574 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.240119 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8" Oct 30 12:58:48.241802 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.241780 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" event={"ID":"4519e0b9-67ff-4ac5-9da9-af1c10707a29","Type":"ContainerStarted","Data":"8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e"} Oct 30 12:58:48.241905 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.241816 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" event={"ID":"4519e0b9-67ff-4ac5-9da9-af1c10707a29","Type":"ContainerStarted","Data":"a4474dfcc9e9b6909285900de3d54f14e0cf8c5d022527dc93eb36171e761fbc"} Oct 30 12:58:48.241905 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.241863 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:58:48.244330 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.244295 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" event={"ID":"1498d838-1fae-4b00-aaff-1adba5a4614f","Type":"ContainerStarted","Data":"d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1"} Oct 30 12:58:48.244413 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.244358 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:48.244413 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.244376 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" event={"ID":"1498d838-1fae-4b00-aaff-1adba5a4614f","Type":"ContainerStarted","Data":"abba3650193426407735088e383a83e427076223eb7a43d60e162feefb579da7"} Oct 30 12:58:48.245390 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.245369 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" podUID="fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" containerName="manager" containerID="cri-o://5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0" gracePeriod=2 Oct 30 12:58:48.302268 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.302215 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8" podStartSLOduration=2.30220028 podStartE2EDuration="2.30220028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:48.300990629 +0000 UTC m=+435.116185053" watchObservedRunningTime="2026-10-30 12:58:48.30220028 +0000 UTC m=+435.117394703" Oct 30 12:58:48.359686 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.359631 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" podStartSLOduration=2.359618081 podStartE2EDuration="2.359618081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:48.357291326 +0000 UTC m=+435.172485749" watchObservedRunningTime="2026-10-30 12:58:48.359618081 +0000 UTC m=+435.174812503" Oct 30 12:58:48.408862 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.408540 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" podStartSLOduration=2.408520754 podStartE2EDuration="2.408520754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:48.406793444 +0000 UTC m=+435.221987868" watchObservedRunningTime="2026-10-30 12:58:48.408520754 +0000 UTC m=+435.223715179" Oct 30 12:58:48.515075 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.515052 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:48.526820 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:48.526791 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-858bb878d8-cpc98\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:48.575717 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.575686 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-extensions-socket-volume\") pod \"fa36ac34-1cd2-44fd-b4b7-f8483469b2c6\" (UID: \"fa36ac34-1cd2-44fd-b4b7-f8483469b2c6\") " Oct 30 12:58:48.575901 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.575815 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-kube-api-access-bm6k2\" (UniqueName: \"kubernetes.io/projected/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-kube-api-access-bm6k2\") pod \"fa36ac34-1cd2-44fd-b4b7-f8483469b2c6\" (UID: \"fa36ac34-1cd2-44fd-b4b7-f8483469b2c6\") " Oct 30 12:58:48.576066 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.576042 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-extensions-socket-volume" pod "fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" (UID: "fa36ac34-1cd2-44fd-b4b7-f8483469b2c6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:48.577853 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.577834 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-kube-api-access-bm6k2" pod "fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" (UID: "fa36ac34-1cd2-44fd-b4b7-f8483469b2c6"). InnerVolumeSpecName "kube-api-access-bm6k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:58:48.676522 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.676481 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm6k2\" (UniqueName: \"kubernetes.io/projected/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-kube-api-access-bm6k2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:58:48.676522 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:48.676512 5049 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6-extensions-socket-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:58:49.251678 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:49.251645 5049 generic.go:356] "Generic (PLEG): container finished" podID="fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" containerID="5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0" exitCode=0 Oct 30 12:58:49.252145 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:49.251690 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:49.252145 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:49.251743 5049 scope.go:122] "RemoveContainer" containerID="5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0" Oct 30 12:58:49.260474 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:49.260456 5049 scope.go:122] "RemoveContainer" containerID="5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0" Oct 30 12:58:49.260719 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:49.260698 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0\": container with ID starting with 5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0 not found: ID does not exist" containerID="5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0" Oct 30 12:58:49.260767 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:49.260727 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0"} err="failed to get container status \"5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0\": rpc error: code = NotFound desc = could not find container \"5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0\": container with ID starting with 5332c2505daa22dbefd8ecb578641b94719fedaca7a67ebd2d306f984e94ece0 not found: ID does not exist" Oct 30 12:58:49.264372 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:49.264348 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-858bb878d8-cpc98\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:49.275839 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:49.275810 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-858bb878d8-cpc98\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-cpc98" Oct 30 12:58:49.332912 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:49.332886 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d7a2d9-fe9f-4272-ae32-07ac0f646bde" path="/var/lib/kubelet/pods/e9d7a2d9-fe9f-4272-ae32-07ac0f646bde/volumes" Oct 30 12:58:49.333221 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:49.333209 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa36ac34-1cd2-44fd-b4b7-f8483469b2c6" path="/var/lib/kubelet/pods/fa36ac34-1cd2-44fd-b4b7-f8483469b2c6/volumes" Oct 30 12:58:56.676179 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:58:56.676136 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:58:59.254179 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.254147 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:59.254542 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.254340 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:58:59.254542 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.254384 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mv4d8" Oct 30 12:58:59.478034 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.478000 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl"] Oct 30 12:58:59.478301 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.478243 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" podUID="1498d838-1fae-4b00-aaff-1adba5a4614f" containerName="manager" containerID="cri-o://d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1" gracePeriod=10 Oct 30 12:58:59.731236 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.731207 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:58:59.736890 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.736864 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87"] Oct 30 12:58:59.737249 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.737232 5049 cpu_manager.go:414] "containerMap: removing container" podUID="1498d838-1fae-4b00-aaff-1adba5a4614f" containerName="manager" Oct 30 12:58:59.737350 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.737251 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1498d838-1fae-4b00-aaff-1adba5a4614f" containerName="manager" Oct 30 12:58:59.737412 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.737394 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="1498d838-1fae-4b00-aaff-1adba5a4614f" containerName="manager" Oct 30 12:58:59.740095 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.740069 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 12:58:59.772891 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.772858 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87"] Oct 30 12:58:59.872295 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.872204 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1498d838-1fae-4b00-aaff-1adba5a4614f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1498d838-1fae-4b00-aaff-1adba5a4614f-extensions-socket-volume\") pod \"1498d838-1fae-4b00-aaff-1adba5a4614f\" (UID: \"1498d838-1fae-4b00-aaff-1adba5a4614f\") " Oct 30 12:58:59.872295 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.872273 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1498d838-1fae-4b00-aaff-1adba5a4614f-kube-api-access-vh75l\" (UniqueName: \"kubernetes.io/projected/1498d838-1fae-4b00-aaff-1adba5a4614f-kube-api-access-vh75l\") pod \"1498d838-1fae-4b00-aaff-1adba5a4614f\" (UID: \"1498d838-1fae-4b00-aaff-1adba5a4614f\") " Oct 30 12:58:59.872558 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.872429 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlv87\" (UID: \"dae44be0-4edf-40eb-9ec1-bc20e5a8de75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 12:58:59.872606 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.872565 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvzg\" (UniqueName: \"kubernetes.io/projected/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-kube-api-access-4rvzg\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlv87\" (UID: \"dae44be0-4edf-40eb-9ec1-bc20e5a8de75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 12:58:59.872643 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.872599 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1498d838-1fae-4b00-aaff-1adba5a4614f-extensions-socket-volume" pod "1498d838-1fae-4b00-aaff-1adba5a4614f" (UID: "1498d838-1fae-4b00-aaff-1adba5a4614f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:59.874440 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.874410 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1498d838-1fae-4b00-aaff-1adba5a4614f-kube-api-access-vh75l" pod "1498d838-1fae-4b00-aaff-1adba5a4614f" (UID: "1498d838-1fae-4b00-aaff-1adba5a4614f"). InnerVolumeSpecName "kube-api-access-vh75l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:58:59.973428 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.973387 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvzg\" (UniqueName: \"kubernetes.io/projected/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-kube-api-access-4rvzg\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlv87\" (UID: \"dae44be0-4edf-40eb-9ec1-bc20e5a8de75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 12:58:59.973579 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.973460 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlv87\" (UID: \"dae44be0-4edf-40eb-9ec1-bc20e5a8de75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 12:58:59.973579 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.973547 5049 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1498d838-1fae-4b00-aaff-1adba5a4614f-extensions-socket-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:58:59.973579 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.973562 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh75l\" (UniqueName: \"kubernetes.io/projected/1498d838-1fae-4b00-aaff-1adba5a4614f-kube-api-access-vh75l\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:58:59.973874 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.973853 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlv87\" (UID: \"dae44be0-4edf-40eb-9ec1-bc20e5a8de75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 12:58:59.998858 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:58:59.998829 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvzg\" (UniqueName: \"kubernetes.io/projected/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-kube-api-access-4rvzg\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mlv87\" (UID: \"dae44be0-4edf-40eb-9ec1-bc20e5a8de75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 12:59:00.053305 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.053266 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 12:59:00.310644 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.310609 5049 generic.go:356] "Generic (PLEG): container finished" podID="1498d838-1fae-4b00-aaff-1adba5a4614f" containerID="d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1" exitCode=0 Oct 30 12:59:00.311006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.310658 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" Oct 30 12:59:00.311006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.310680 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" event={"ID":"1498d838-1fae-4b00-aaff-1adba5a4614f","Type":"ContainerDied","Data":"d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1"} Oct 30 12:59:00.311006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.310734 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl" event={"ID":"1498d838-1fae-4b00-aaff-1adba5a4614f","Type":"ContainerDied","Data":"abba3650193426407735088e383a83e427076223eb7a43d60e162feefb579da7"} Oct 30 12:59:00.311006 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.310757 5049 scope.go:122] "RemoveContainer" containerID="d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1" Oct 30 12:59:00.318964 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.318947 5049 scope.go:122] "RemoveContainer" containerID="d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1" Oct 30 12:59:00.319230 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:59:00.319214 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1\": container with ID starting with d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1 not found: ID does not exist" containerID="d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1" Oct 30 12:59:00.319277 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.319238 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1"} err="failed to get container status \"d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1\": rpc error: code = NotFound desc = could not find container \"d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1\": container with ID starting with d6d0ba867d173cebc2ee4d1ccd0747b5259a1b37a3a8cdcfa6636a4cbe2249b1 not found: ID does not exist" Oct 30 12:59:00.369276 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.369246 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl"] Oct 30 12:59:00.375405 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.375381 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-rd5sl"] Oct 30 12:59:00.382897 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:00.382878 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87"] Oct 30 12:59:00.386056 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:59:00.386028 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae44be0_4edf_40eb_9ec1_bc20e5a8de75.slice/crio-b5e6c27fa3cc1d38c5b551bdf9985c7bf0afba4ecf4431a5cedef2c2d32ebc72 WatchSource:0}: Error finding container b5e6c27fa3cc1d38c5b551bdf9985c7bf0afba4ecf4431a5cedef2c2d32ebc72: Status 404 returned error can't find the container with id b5e6c27fa3cc1d38c5b551bdf9985c7bf0afba4ecf4431a5cedef2c2d32ebc72 Oct 30 12:59:01.317004 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:01.316959 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" event={"ID":"dae44be0-4edf-40eb-9ec1-bc20e5a8de75","Type":"ContainerStarted","Data":"a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84"} Oct 30 12:59:01.317463 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:01.317070 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 12:59:01.317463 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:01.317100 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" event={"ID":"dae44be0-4edf-40eb-9ec1-bc20e5a8de75","Type":"ContainerStarted","Data":"b5e6c27fa3cc1d38c5b551bdf9985c7bf0afba4ecf4431a5cedef2c2d32ebc72"} Oct 30 12:59:01.333373 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:01.333344 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1498d838-1fae-4b00-aaff-1adba5a4614f" path="/var/lib/kubelet/pods/1498d838-1fae-4b00-aaff-1adba5a4614f/volumes" Oct 30 12:59:01.378692 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:01.378648 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" podStartSLOduration=2.378636937 podStartE2EDuration="2.378636937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:01.378083246 +0000 UTC m=+448.193277668" watchObservedRunningTime="2026-10-30 12:59:01.378636937 +0000 UTC m=+448.193831360" Oct 30 12:59:12.324725 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:12.324692 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 12:59:12.430893 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:12.430858 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h"] Oct 30 12:59:12.431085 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:12.431060 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" podUID="4519e0b9-67ff-4ac5-9da9-af1c10707a29" containerName="manager" containerID="cri-o://8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e" gracePeriod=10 Oct 30 12:59:12.682630 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:12.682606 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:59:12.780854 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:12.780823 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4519e0b9-67ff-4ac5-9da9-af1c10707a29-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4519e0b9-67ff-4ac5-9da9-af1c10707a29-extensions-socket-volume\") pod \"4519e0b9-67ff-4ac5-9da9-af1c10707a29\" (UID: \"4519e0b9-67ff-4ac5-9da9-af1c10707a29\") " Oct 30 12:59:12.781040 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:12.780875 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4519e0b9-67ff-4ac5-9da9-af1c10707a29-kube-api-access-2bhzc\" (UniqueName: \"kubernetes.io/projected/4519e0b9-67ff-4ac5-9da9-af1c10707a29-kube-api-access-2bhzc\") pod \"4519e0b9-67ff-4ac5-9da9-af1c10707a29\" (UID: \"4519e0b9-67ff-4ac5-9da9-af1c10707a29\") " Oct 30 12:59:12.781223 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:12.781196 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4519e0b9-67ff-4ac5-9da9-af1c10707a29-extensions-socket-volume" pod "4519e0b9-67ff-4ac5-9da9-af1c10707a29" (UID: "4519e0b9-67ff-4ac5-9da9-af1c10707a29"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:59:12.782901 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:12.782874 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4519e0b9-67ff-4ac5-9da9-af1c10707a29-kube-api-access-2bhzc" pod "4519e0b9-67ff-4ac5-9da9-af1c10707a29" (UID: "4519e0b9-67ff-4ac5-9da9-af1c10707a29"). InnerVolumeSpecName "kube-api-access-2bhzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:59:12.881755 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:12.881691 5049 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4519e0b9-67ff-4ac5-9da9-af1c10707a29-extensions-socket-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:59:12.881755 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:12.881715 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bhzc\" (UniqueName: \"kubernetes.io/projected/4519e0b9-67ff-4ac5-9da9-af1c10707a29-kube-api-access-2bhzc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:59:13.370994 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:13.370966 5049 generic.go:356] "Generic (PLEG): container finished" podID="4519e0b9-67ff-4ac5-9da9-af1c10707a29" containerID="8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e" exitCode=0 Oct 30 12:59:13.371389 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:13.371015 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" Oct 30 12:59:13.371389 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:13.371052 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" event={"ID":"4519e0b9-67ff-4ac5-9da9-af1c10707a29","Type":"ContainerDied","Data":"8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e"} Oct 30 12:59:13.371389 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:13.371102 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h" event={"ID":"4519e0b9-67ff-4ac5-9da9-af1c10707a29","Type":"ContainerDied","Data":"a4474dfcc9e9b6909285900de3d54f14e0cf8c5d022527dc93eb36171e761fbc"} Oct 30 12:59:13.371389 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:13.371127 5049 scope.go:122] "RemoveContainer" containerID="8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e" Oct 30 12:59:13.379362 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:13.379346 5049 scope.go:122] "RemoveContainer" containerID="8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e" Oct 30 12:59:13.379630 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:59:13.379610 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e\": container with ID starting with 8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e not found: ID does not exist" containerID="8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e" Oct 30 12:59:13.379691 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:13.379643 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e"} err="failed to get container status \"8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e\": rpc error: code = NotFound desc = could not find container \"8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e\": container with ID starting with 8c76901b254449558e4db7431bf5a7bcf681ce0186ec6f5830bf9e4d6dbbb09e not found: ID does not exist" Oct 30 12:59:13.416942 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:13.416906 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h"] Oct 30 12:59:13.435180 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:13.435152 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-858bb878d8-7gf8h"] Oct 30 12:59:15.334153 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:15.334120 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4519e0b9-67ff-4ac5-9da9-af1c10707a29" path="/var/lib/kubelet/pods/4519e0b9-67ff-4ac5-9da9-af1c10707a29/volumes" Oct 30 12:59:16.859435 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:16.859395 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:17.065042 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:17.065010 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:23.888283 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:23.888247 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:26.675094 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:59:26.675055 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 12:59:31.646658 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:31.646620 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:31.837992 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:31.837958 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:32.481846 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.481810 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4xbv5"] Oct 30 12:59:32.482268 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.482247 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4519e0b9-67ff-4ac5-9da9-af1c10707a29" containerName="manager" Oct 30 12:59:32.482383 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.482269 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4519e0b9-67ff-4ac5-9da9-af1c10707a29" containerName="manager" Oct 30 12:59:32.482383 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.482369 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4519e0b9-67ff-4ac5-9da9-af1c10707a29" containerName="manager" Oct 30 12:59:32.486704 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.486681 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4xbv5" Oct 30 12:59:32.503785 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.503760 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gjrjz\"" Oct 30 12:59:32.512139 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.512114 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4xbv5"] Oct 30 12:59:32.531312 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.531288 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzzrm\" (UniqueName: \"kubernetes.io/projected/3d13c652-82c5-454d-9a89-30cd97b4fdff-kube-api-access-rzzrm\") pod \"authorino-57f7f8659f-4xbv5\" (UID: \"3d13c652-82c5-454d-9a89-30cd97b4fdff\") " pod="kuadrant-system/authorino-57f7f8659f-4xbv5" Oct 30 12:59:32.628306 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.628274 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:32.632405 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.632386 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzzrm\" (UniqueName: \"kubernetes.io/projected/3d13c652-82c5-454d-9a89-30cd97b4fdff-kube-api-access-rzzrm\") pod \"authorino-57f7f8659f-4xbv5\" (UID: \"3d13c652-82c5-454d-9a89-30cd97b4fdff\") " pod="kuadrant-system/authorino-57f7f8659f-4xbv5" Oct 30 12:59:32.654399 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.654367 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzzrm\" (UniqueName: \"kubernetes.io/projected/3d13c652-82c5-454d-9a89-30cd97b4fdff-kube-api-access-rzzrm\") pod \"authorino-57f7f8659f-4xbv5\" (UID: \"3d13c652-82c5-454d-9a89-30cd97b4fdff\") " pod="kuadrant-system/authorino-57f7f8659f-4xbv5" Oct 30 12:59:32.787924 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.787846 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:32.797856 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.797824 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4xbv5" Oct 30 12:59:32.982926 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.982891 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9lb2r"] Oct 30 12:59:32.987450 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:32.987429 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" Oct 30 12:59:33.025714 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:33.025683 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9lb2r"] Oct 30 12:59:33.035617 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:33.035589 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brr2t\" (UniqueName: \"kubernetes.io/projected/2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2-kube-api-access-brr2t\") pod \"authorino-56c6c9ddd5-9lb2r\" (UID: \"2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2\") " pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" Oct 30 12:59:33.128168 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:33.128136 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4xbv5"] Oct 30 12:59:33.131663 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:59:33.131634 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d13c652_82c5_454d_9a89_30cd97b4fdff.slice/crio-b812be7479e3efcdb7a36dbd960bdb719a21e5fa5f9aa95bd0ffac149469a1fb WatchSource:0}: Error finding container b812be7479e3efcdb7a36dbd960bdb719a21e5fa5f9aa95bd0ffac149469a1fb: Status 404 returned error can't find the container with id b812be7479e3efcdb7a36dbd960bdb719a21e5fa5f9aa95bd0ffac149469a1fb Oct 30 12:59:33.136039 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:33.136008 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brr2t\" (UniqueName: \"kubernetes.io/projected/2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2-kube-api-access-brr2t\") pod \"authorino-56c6c9ddd5-9lb2r\" (UID: \"2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2\") " pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" Oct 30 12:59:33.163527 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:33.163497 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brr2t\" (UniqueName: \"kubernetes.io/projected/2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2-kube-api-access-brr2t\") pod \"authorino-56c6c9ddd5-9lb2r\" (UID: \"2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2\") " pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" Oct 30 12:59:33.302953 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:33.302922 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" Oct 30 12:59:33.462732 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:33.462650 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4xbv5" event={"ID":"3d13c652-82c5-454d-9a89-30cd97b4fdff","Type":"ContainerStarted","Data":"b812be7479e3efcdb7a36dbd960bdb719a21e5fa5f9aa95bd0ffac149469a1fb"} Oct 30 12:59:33.597608 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:33.597579 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9lb2r"] Oct 30 12:59:33.600750 ip-10-0-133-61 kubenswrapper[5049]: W1030 12:59:33.600723 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fecc12f_79b3_4dd5_ad34_2f5ab2f8b2f2.slice/crio-ed55cf9af10b763f0d04990e0007eeb0e23bda6ddea49cf2248b8beafd9e657b WatchSource:0}: Error finding container ed55cf9af10b763f0d04990e0007eeb0e23bda6ddea49cf2248b8beafd9e657b: Status 404 returned error can't find the container with id ed55cf9af10b763f0d04990e0007eeb0e23bda6ddea49cf2248b8beafd9e657b Oct 30 12:59:34.470009 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:34.469968 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" event={"ID":"2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2","Type":"ContainerStarted","Data":"ed55cf9af10b763f0d04990e0007eeb0e23bda6ddea49cf2248b8beafd9e657b"} Oct 30 12:59:35.476073 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:35.476032 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4xbv5" event={"ID":"3d13c652-82c5-454d-9a89-30cd97b4fdff","Type":"ContainerStarted","Data":"7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c"} Oct 30 12:59:35.477555 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:35.477531 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" event={"ID":"2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2","Type":"ContainerStarted","Data":"35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f"} Oct 30 12:59:35.511501 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:35.511453 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4xbv5" podStartSLOduration=1.900807308 podStartE2EDuration="3.511439012s" totalImagesPullingTime="1.610631704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:35.510285971 +0000 UTC m=+482.325480395" watchObservedRunningTime="2026-10-30 12:59:35.511439012 +0000 UTC m=+482.326633434" Oct 30 12:59:35.546885 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:35.546831 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" podStartSLOduration=2.168992084 podStartE2EDuration="3.546817522s" totalImagesPullingTime="1.377825438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:35.54580986 +0000 UTC m=+482.361004284" watchObservedRunningTime="2026-10-30 12:59:35.546817522 +0000 UTC m=+482.362011942" Oct 30 12:59:35.574367 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:35.574339 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4xbv5"] Oct 30 12:59:37.485944 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:37.485909 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4xbv5" podUID="3d13c652-82c5-454d-9a89-30cd97b4fdff" containerName="authorino" containerID="cri-o://7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c" gracePeriod=30 Oct 30 12:59:37.731213 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:37.731186 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4xbv5" Oct 30 12:59:37.774648 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:37.774592 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d13c652-82c5-454d-9a89-30cd97b4fdff-kube-api-access-rzzrm\" (UniqueName: \"kubernetes.io/projected/3d13c652-82c5-454d-9a89-30cd97b4fdff-kube-api-access-rzzrm\") pod \"3d13c652-82c5-454d-9a89-30cd97b4fdff\" (UID: \"3d13c652-82c5-454d-9a89-30cd97b4fdff\") " Oct 30 12:59:37.776963 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:37.776938 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d13c652-82c5-454d-9a89-30cd97b4fdff-kube-api-access-rzzrm" pod "3d13c652-82c5-454d-9a89-30cd97b4fdff" (UID: "3d13c652-82c5-454d-9a89-30cd97b4fdff"). InnerVolumeSpecName "kube-api-access-rzzrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:59:37.875246 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:37.875216 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzzrm\" (UniqueName: \"kubernetes.io/projected/3d13c652-82c5-454d-9a89-30cd97b4fdff-kube-api-access-rzzrm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 12:59:38.491151 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:38.491119 5049 generic.go:356] "Generic (PLEG): container finished" podID="3d13c652-82c5-454d-9a89-30cd97b4fdff" containerID="7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c" exitCode=0 Oct 30 12:59:38.491590 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:38.491166 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4xbv5" Oct 30 12:59:38.491590 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:38.491166 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4xbv5" event={"ID":"3d13c652-82c5-454d-9a89-30cd97b4fdff","Type":"ContainerDied","Data":"7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c"} Oct 30 12:59:38.491590 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:38.491209 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4xbv5" event={"ID":"3d13c652-82c5-454d-9a89-30cd97b4fdff","Type":"ContainerDied","Data":"b812be7479e3efcdb7a36dbd960bdb719a21e5fa5f9aa95bd0ffac149469a1fb"} Oct 30 12:59:38.491590 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:38.491226 5049 scope.go:122] "RemoveContainer" containerID="7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c" Oct 30 12:59:38.500254 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:38.500237 5049 scope.go:122] "RemoveContainer" containerID="7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c" Oct 30 12:59:38.500555 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:59:38.500533 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c\": container with ID starting with 7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c not found: ID does not exist" containerID="7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c" Oct 30 12:59:38.500640 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:38.500567 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c"} err="failed to get container status \"7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c\": rpc error: code = NotFound desc = could not find container \"7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c\": container with ID starting with 7b611726209eeca63433be29d749bf5ab17622e0daf952eba3b2a96291542c3c not found: ID does not exist" Oct 30 12:59:38.535480 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:38.535456 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4xbv5"] Oct 30 12:59:38.545451 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:38.545425 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4xbv5"] Oct 30 12:59:39.333522 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:39.333493 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d13c652-82c5-454d-9a89-30cd97b4fdff" path="/var/lib/kubelet/pods/3d13c652-82c5-454d-9a89-30cd97b4fdff/volumes" Oct 30 12:59:45.837814 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:45.837727 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:46.000215 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:46.000179 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:51.556104 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:51.556072 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:51.714655 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:51.714622 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:52.338890 ip-10-0-133-61 kubenswrapper[5049]: I1030 12:59:52.338856 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:56.676004 ip-10-0-133-61 kubenswrapper[5049]: E1030 12:59:56.675964 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:00:01.859346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:01.859297 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:04.914114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:04.914082 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:05.094387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.094350 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:05.179963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.179883 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zckm8"] Oct 30 13:00:05.180362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.180288 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3d13c652-82c5-454d-9a89-30cd97b4fdff" containerName="authorino" Oct 30 13:00:05.180362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.180307 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d13c652-82c5-454d-9a89-30cd97b4fdff" containerName="authorino" Oct 30 13:00:05.180510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.180418 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d13c652-82c5-454d-9a89-30cd97b4fdff" containerName="authorino" Oct 30 13:00:05.183540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.183516 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zckm8" Oct 30 13:00:05.206205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.206182 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zckm8"] Oct 30 13:00:05.312995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.312960 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfnq\" (UniqueName: \"kubernetes.io/projected/22f64431-1568-41a5-ac3d-b0bf3019fede-kube-api-access-nmfnq\") pod \"authorino-6f6944bcdd-zckm8\" (UID: \"22f64431-1568-41a5-ac3d-b0bf3019fede\") " pod="kuadrant-system/authorino-6f6944bcdd-zckm8" Oct 30 13:00:05.387458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.387424 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zckm8"] Oct 30 13:00:05.387704 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:00:05.387682 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nmfnq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-zckm8" podUID="22f64431-1568-41a5-ac3d-b0bf3019fede" Oct 30 13:00:05.413865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.413832 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfnq\" (UniqueName: \"kubernetes.io/projected/22f64431-1568-41a5-ac3d-b0bf3019fede-kube-api-access-nmfnq\") pod \"authorino-6f6944bcdd-zckm8\" (UID: \"22f64431-1568-41a5-ac3d-b0bf3019fede\") " pod="kuadrant-system/authorino-6f6944bcdd-zckm8" Oct 30 13:00:05.419358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.419312 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-pplvl"] Oct 30 13:00:05.423800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.423783 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-pplvl" Oct 30 13:00:05.439680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.439649 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 13:00:05.439787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.439754 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfnq\" (UniqueName: \"kubernetes.io/projected/22f64431-1568-41a5-ac3d-b0bf3019fede-kube-api-access-nmfnq\") pod \"authorino-6f6944bcdd-zckm8\" (UID: \"22f64431-1568-41a5-ac3d-b0bf3019fede\") " pod="kuadrant-system/authorino-6f6944bcdd-zckm8" Oct 30 13:00:05.443737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.443716 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-pplvl"] Oct 30 13:00:05.513693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.513661 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-pplvl"] Oct 30 13:00:05.513888 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:00:05.513870 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-m8z8h tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-pplvl" podUID="2e1ae449-527e-445a-9d81-9d319d8fe86b" Oct 30 13:00:05.514181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.514161 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e1ae449-527e-445a-9d81-9d319d8fe86b-tls-cert\") pod \"authorino-6889cbd8c4-pplvl\" (UID: \"2e1ae449-527e-445a-9d81-9d319d8fe86b\") " pod="kuadrant-system/authorino-6889cbd8c4-pplvl" Oct 30 13:00:05.514266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.514210 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8z8h\" (UniqueName: \"kubernetes.io/projected/2e1ae449-527e-445a-9d81-9d319d8fe86b-kube-api-access-m8z8h\") pod \"authorino-6889cbd8c4-pplvl\" (UID: \"2e1ae449-527e-445a-9d81-9d319d8fe86b\") " pod="kuadrant-system/authorino-6889cbd8c4-pplvl" Oct 30 13:00:05.541005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.540976 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-86587766db-n2bh7"] Oct 30 13:00:05.545766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.545745 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86587766db-n2bh7" Oct 30 13:00:05.570196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.570169 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86587766db-n2bh7"] Oct 30 13:00:05.614712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.614684 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-pplvl" Oct 30 13:00:05.614712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.614708 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zckm8" Oct 30 13:00:05.614960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.614942 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e1ae449-527e-445a-9d81-9d319d8fe86b-tls-cert\") pod \"authorino-6889cbd8c4-pplvl\" (UID: \"2e1ae449-527e-445a-9d81-9d319d8fe86b\") " pod="kuadrant-system/authorino-6889cbd8c4-pplvl" Oct 30 13:00:05.615002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.614990 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8z8h\" (UniqueName: \"kubernetes.io/projected/2e1ae449-527e-445a-9d81-9d319d8fe86b-kube-api-access-m8z8h\") pod \"authorino-6889cbd8c4-pplvl\" (UID: \"2e1ae449-527e-445a-9d81-9d319d8fe86b\") " pod="kuadrant-system/authorino-6889cbd8c4-pplvl" Oct 30 13:00:05.617504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.617483 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e1ae449-527e-445a-9d81-9d319d8fe86b-tls-cert\") pod \"authorino-6889cbd8c4-pplvl\" (UID: \"2e1ae449-527e-445a-9d81-9d319d8fe86b\") " pod="kuadrant-system/authorino-6889cbd8c4-pplvl" Oct 30 13:00:05.630205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.630180 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-pplvl" Oct 30 13:00:05.633015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.632999 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zckm8" Oct 30 13:00:05.636180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.636158 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8z8h\" (UniqueName: \"kubernetes.io/projected/2e1ae449-527e-445a-9d81-9d319d8fe86b-kube-api-access-m8z8h\") pod \"authorino-6889cbd8c4-pplvl\" (UID: \"2e1ae449-527e-445a-9d81-9d319d8fe86b\") " pod="kuadrant-system/authorino-6889cbd8c4-pplvl" Oct 30 13:00:05.715624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.715551 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e1ae449-527e-445a-9d81-9d319d8fe86b-tls-cert\" (UniqueName: \"kubernetes.io/secret/2e1ae449-527e-445a-9d81-9d319d8fe86b-tls-cert\") pod \"2e1ae449-527e-445a-9d81-9d319d8fe86b\" (UID: \"2e1ae449-527e-445a-9d81-9d319d8fe86b\") " Oct 30 13:00:05.715624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.715604 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f64431-1568-41a5-ac3d-b0bf3019fede-kube-api-access-nmfnq\" (UniqueName: \"kubernetes.io/projected/22f64431-1568-41a5-ac3d-b0bf3019fede-kube-api-access-nmfnq\") pod \"22f64431-1568-41a5-ac3d-b0bf3019fede\" (UID: \"22f64431-1568-41a5-ac3d-b0bf3019fede\") " Oct 30 13:00:05.715802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.715724 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29b1f0bb-60a8-45e8-910b-284834a48f47-tls-cert\") pod \"authorino-86587766db-n2bh7\" (UID: \"29b1f0bb-60a8-45e8-910b-284834a48f47\") " pod="kuadrant-system/authorino-86587766db-n2bh7" Oct 30 13:00:05.715802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.715770 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shbw2\" (UniqueName: \"kubernetes.io/projected/29b1f0bb-60a8-45e8-910b-284834a48f47-kube-api-access-shbw2\") pod \"authorino-86587766db-n2bh7\" (UID: \"29b1f0bb-60a8-45e8-910b-284834a48f47\") " pod="kuadrant-system/authorino-86587766db-n2bh7" Oct 30 13:00:05.717705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.717676 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1ae449-527e-445a-9d81-9d319d8fe86b-tls-cert" pod "2e1ae449-527e-445a-9d81-9d319d8fe86b" (UID: "2e1ae449-527e-445a-9d81-9d319d8fe86b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:00:05.717813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.717677 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f64431-1568-41a5-ac3d-b0bf3019fede-kube-api-access-nmfnq" pod "22f64431-1568-41a5-ac3d-b0bf3019fede" (UID: "22f64431-1568-41a5-ac3d-b0bf3019fede"). InnerVolumeSpecName "kube-api-access-nmfnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:05.816669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.816642 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1ae449-527e-445a-9d81-9d319d8fe86b-kube-api-access-m8z8h\" (UniqueName: \"kubernetes.io/projected/2e1ae449-527e-445a-9d81-9d319d8fe86b-kube-api-access-m8z8h\") pod \"2e1ae449-527e-445a-9d81-9d319d8fe86b\" (UID: \"2e1ae449-527e-445a-9d81-9d319d8fe86b\") " Oct 30 13:00:05.816829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.816737 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29b1f0bb-60a8-45e8-910b-284834a48f47-tls-cert\") pod \"authorino-86587766db-n2bh7\" (UID: \"29b1f0bb-60a8-45e8-910b-284834a48f47\") " pod="kuadrant-system/authorino-86587766db-n2bh7" Oct 30 13:00:05.816829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.816815 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shbw2\" (UniqueName: \"kubernetes.io/projected/29b1f0bb-60a8-45e8-910b-284834a48f47-kube-api-access-shbw2\") pod \"authorino-86587766db-n2bh7\" (UID: \"29b1f0bb-60a8-45e8-910b-284834a48f47\") " pod="kuadrant-system/authorino-86587766db-n2bh7" Oct 30 13:00:05.816930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.816881 5049 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e1ae449-527e-445a-9d81-9d319d8fe86b-tls-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:00:05.816930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.816895 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmfnq\" (UniqueName: \"kubernetes.io/projected/22f64431-1568-41a5-ac3d-b0bf3019fede-kube-api-access-nmfnq\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:00:05.818755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.818729 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1ae449-527e-445a-9d81-9d319d8fe86b-kube-api-access-m8z8h" pod "2e1ae449-527e-445a-9d81-9d319d8fe86b" (UID: "2e1ae449-527e-445a-9d81-9d319d8fe86b"). InnerVolumeSpecName "kube-api-access-m8z8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:05.819066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.819047 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29b1f0bb-60a8-45e8-910b-284834a48f47-tls-cert\") pod \"authorino-86587766db-n2bh7\" (UID: \"29b1f0bb-60a8-45e8-910b-284834a48f47\") " pod="kuadrant-system/authorino-86587766db-n2bh7" Oct 30 13:00:05.838194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.838162 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shbw2\" (UniqueName: \"kubernetes.io/projected/29b1f0bb-60a8-45e8-910b-284834a48f47-kube-api-access-shbw2\") pod \"authorino-86587766db-n2bh7\" (UID: \"29b1f0bb-60a8-45e8-910b-284834a48f47\") " pod="kuadrant-system/authorino-86587766db-n2bh7" Oct 30 13:00:05.857990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.857965 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86587766db-n2bh7" Oct 30 13:00:05.918148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:05.918107 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8z8h\" (UniqueName: \"kubernetes.io/projected/2e1ae449-527e-445a-9d81-9d319d8fe86b-kube-api-access-m8z8h\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:00:06.150801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.150766 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86587766db-n2bh7"] Oct 30 13:00:06.153704 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:00:06.153678 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29b1f0bb_60a8_45e8_910b_284834a48f47.slice/crio-b143abf1df5349152fb3c5746d7011518318e7566d65b0fecde8e042365689c4 WatchSource:0}: Error finding container b143abf1df5349152fb3c5746d7011518318e7566d65b0fecde8e042365689c4: Status 404 returned error can't find the container with id b143abf1df5349152fb3c5746d7011518318e7566d65b0fecde8e042365689c4 Oct 30 13:00:06.620170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.620137 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zckm8" Oct 30 13:00:06.620170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.620153 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86587766db-n2bh7" event={"ID":"29b1f0bb-60a8-45e8-910b-284834a48f47","Type":"ContainerStarted","Data":"cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348"} Oct 30 13:00:06.620493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.620193 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86587766db-n2bh7" event={"ID":"29b1f0bb-60a8-45e8-910b-284834a48f47","Type":"ContainerStarted","Data":"b143abf1df5349152fb3c5746d7011518318e7566d65b0fecde8e042365689c4"} Oct 30 13:00:06.620493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.620299 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-pplvl" Oct 30 13:00:06.657711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.657665 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-86587766db-n2bh7" podStartSLOduration=1.31245702 podStartE2EDuration="1.657651486s" totalImagesPullingTime="345.194466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:06.655649772 +0000 UTC m=+513.470844196" watchObservedRunningTime="2026-10-30 13:00:06.657651486 +0000 UTC m=+513.472845908" Oct 30 13:00:06.688432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.688402 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9lb2r"] Oct 30 13:00:06.688605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.688579 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" podUID="2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2" containerName="authorino" containerID="cri-o://35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f" gracePeriod=30 Oct 30 13:00:06.732991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.732958 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-pplvl"] Oct 30 13:00:06.737302 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.737272 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-pplvl"] Oct 30 13:00:06.791848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.791818 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zckm8"] Oct 30 13:00:06.796606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.796574 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zckm8"] Oct 30 13:00:06.923295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:06.923269 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" Oct 30 13:00:07.028837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.028752 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2-kube-api-access-brr2t\" (UniqueName: \"kubernetes.io/projected/2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2-kube-api-access-brr2t\") pod \"2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2\" (UID: \"2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2\") " Oct 30 13:00:07.030827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.030804 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2-kube-api-access-brr2t" pod "2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2" (UID: "2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2"). InnerVolumeSpecName "kube-api-access-brr2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:07.129978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.129944 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brr2t\" (UniqueName: \"kubernetes.io/projected/2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2-kube-api-access-brr2t\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:00:07.334101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.334073 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f64431-1568-41a5-ac3d-b0bf3019fede" path="/var/lib/kubelet/pods/22f64431-1568-41a5-ac3d-b0bf3019fede/volumes" Oct 30 13:00:07.334329 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.334306 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1ae449-527e-445a-9d81-9d319d8fe86b" path="/var/lib/kubelet/pods/2e1ae449-527e-445a-9d81-9d319d8fe86b/volumes" Oct 30 13:00:07.625992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.625892 5049 generic.go:356] "Generic (PLEG): container finished" podID="2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2" containerID="35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f" exitCode=0 Oct 30 13:00:07.625992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.625930 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" Oct 30 13:00:07.625992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.625983 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" event={"ID":"2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2","Type":"ContainerDied","Data":"35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f"} Oct 30 13:00:07.626233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.626020 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9lb2r" event={"ID":"2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2","Type":"ContainerDied","Data":"ed55cf9af10b763f0d04990e0007eeb0e23bda6ddea49cf2248b8beafd9e657b"} Oct 30 13:00:07.626233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.626037 5049 scope.go:122] "RemoveContainer" containerID="35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f" Oct 30 13:00:07.634399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.634379 5049 scope.go:122] "RemoveContainer" containerID="35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f" Oct 30 13:00:07.634668 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:00:07.634649 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f\": container with ID starting with 35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f not found: ID does not exist" containerID="35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f" Oct 30 13:00:07.634718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.634678 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f"} err="failed to get container status \"35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f\": rpc error: code = NotFound desc = could not find container \"35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f\": container with ID starting with 35e031e8ab39c267a421841072bfce6f2d80779718e18c8941a637ebaf5d487f not found: ID does not exist" Oct 30 13:00:07.669719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.669690 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9lb2r"] Oct 30 13:00:07.679636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:07.679612 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9lb2r"] Oct 30 13:00:09.333366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:09.333316 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2" path="/var/lib/kubelet/pods/2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2/volumes" Oct 30 13:00:16.309949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:16.309909 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:16.471514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:16.471480 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:16.630925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:16.630843 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:16.799759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:16.799721 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:20.032103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:20.032066 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:20.341836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:20.341797 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:23.663669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.663636 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k"] Oct 30 13:00:23.664078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.664001 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2" containerName="authorino" Oct 30 13:00:23.664078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.664011 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2" containerName="authorino" Oct 30 13:00:23.664078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.664072 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fecc12f-79b3-4dd5-ad34-2f5ab2f8b2f2" containerName="authorino" Oct 30 13:00:23.667166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.667146 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.697281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.696728 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-ptbtc\"" Oct 30 13:00:23.697281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.696795 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 13:00:23.697281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.696844 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 13:00:23.697281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.696871 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 13:00:23.697668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.697637 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k"] Oct 30 13:00:23.770975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.770937 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.771145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.770992 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.771145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.771127 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.771272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.771247 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qjgb\" (UniqueName: \"kubernetes.io/projected/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-kube-api-access-8qjgb\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.795722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.795688 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:23.872533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.872501 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qjgb\" (UniqueName: \"kubernetes.io/projected/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-kube-api-access-8qjgb\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.872692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.872576 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.872692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.872601 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.872692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.872629 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.873333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.873292 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.873437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.873419 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.875027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.875003 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.893879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.893850 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qjgb\" (UniqueName: \"kubernetes.io/projected/8d3d062c-dc7f-4625-94ad-bb73e9c5244b-kube-api-access-8qjgb\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2kk5k\" (UID: \"8d3d062c-dc7f-4625-94ad-bb73e9c5244b\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:23.948155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.948080 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:23.980572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:23.980541 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:24.115259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:24.115169 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:24.275544 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:24.275512 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k"] Oct 30 13:00:24.278171 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:00:24.278139 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3d062c_dc7f_4625_94ad_bb73e9c5244b.slice/crio-0691d677a2e35c134be6ffc2c38db5a8481c5f8334f9d607ac8f0558f3cc5f08 WatchSource:0}: Error finding container 0691d677a2e35c134be6ffc2c38db5a8481c5f8334f9d607ac8f0558f3cc5f08: Status 404 returned error can't find the container with id 0691d677a2e35c134be6ffc2c38db5a8481c5f8334f9d607ac8f0558f3cc5f08 Oct 30 13:00:24.701954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:24.701923 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" event={"ID":"8d3d062c-dc7f-4625-94ad-bb73e9c5244b","Type":"ContainerStarted","Data":"0691d677a2e35c134be6ffc2c38db5a8481c5f8334f9d607ac8f0558f3cc5f08"} Oct 30 13:00:26.689783 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:00:26.689747 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:00:30.276470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:30.276436 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:30.431343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:30.431293 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:34.759468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:34.759429 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" event={"ID":"8d3d062c-dc7f-4625-94ad-bb73e9c5244b","Type":"ContainerStarted","Data":"f759aec3e2377da56bdbd21af5952516257fb8cc8506ed307f8454e92ff1d2db"} Oct 30 13:00:34.759863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:34.759495 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:34.797416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:34.797369 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" podStartSLOduration=1.844846947 podStartE2EDuration="11.797356664s" totalImagesPullingTime="9.952509717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:34.796786264 +0000 UTC m=+541.611980687" watchObservedRunningTime="2026-10-30 13:00:34.797356664 +0000 UTC m=+541.612551086" Oct 30 13:00:43.209161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:43.209127 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:45.768477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:45.768447 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2kk5k" Oct 30 13:00:54.912041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:54.912008 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:55.126259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:55.126226 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:56.452436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:56.452394 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:56.643228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:56.643191 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:56.677441 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:00:56.677399 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:00:56.821082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:00:56.821045 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:24.138063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:24.137990 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:26.675653 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:01:26.675617 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:01:33.371784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:33.371754 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:01:33.373147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:33.373120 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:01:39.678725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:39.678691 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:39.910983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:39.910951 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:55.059477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.059437 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk"] Oct 30 13:01:55.062953 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.062935 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.087578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.087549 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 13:01:55.087799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.087592 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 13:01:55.087914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.087622 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 13:01:55.087914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.087826 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tdk84\"" Oct 30 13:01:55.090659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.090634 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk"] Oct 30 13:01:55.200354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.200308 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:55.257096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.257060 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9331bf25-1c21-420d-b7cf-fb931432db23-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.257096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.257093 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t68bj\" (UniqueName: \"kubernetes.io/projected/9331bf25-1c21-420d-b7cf-fb931432db23-kube-api-access-t68bj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.257316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.257119 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.257316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.257135 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.257316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.257312 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.257470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.257370 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.358054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.357960 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.358054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.358019 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9331bf25-1c21-420d-b7cf-fb931432db23-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.358054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.358043 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68bj\" (UniqueName: \"kubernetes.io/projected/9331bf25-1c21-420d-b7cf-fb931432db23-kube-api-access-t68bj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.358389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.358073 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.358389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.358093 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.358389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.358159 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.358574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.358546 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.358574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.358563 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.358685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.358661 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.360343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.360297 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9331bf25-1c21-420d-b7cf-fb931432db23-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.360609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.360593 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9331bf25-1c21-420d-b7cf-fb931432db23-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.379981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.379961 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68bj\" (UniqueName: \"kubernetes.io/projected/9331bf25-1c21-420d-b7cf-fb931432db23-kube-api-access-t68bj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk\" (UID: \"9331bf25-1c21-420d-b7cf-fb931432db23\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.679502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.679412 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:55.984956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:55.984867 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk"] Oct 30 13:01:55.988166 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:01:55.988138 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9331bf25_1c21_420d_b7cf_fb931432db23.slice/crio-be14b4b238ece81efa4e0914664074d3e4b4e87fe640755f560791aa47b5123f WatchSource:0}: Error finding container be14b4b238ece81efa4e0914664074d3e4b4e87fe640755f560791aa47b5123f: Status 404 returned error can't find the container with id be14b4b238ece81efa4e0914664074d3e4b4e87fe640755f560791aa47b5123f Oct 30 13:01:56.110852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:56.110807 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" event={"ID":"9331bf25-1c21-420d-b7cf-fb931432db23","Type":"ContainerStarted","Data":"be14b4b238ece81efa4e0914664074d3e4b4e87fe640755f560791aa47b5123f"} Oct 30 13:01:56.696783 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:01:56.696703 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:01:57.067924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:57.067852 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:57.680843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:57.680752 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:58.122898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.122865 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" event={"ID":"9331bf25-1c21-420d-b7cf-fb931432db23","Type":"ContainerStarted","Data":"40d71027498873b16200ad168804c2eef6033441a8345b35927abeb7fc00911b"} Oct 30 13:01:58.162276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.162222 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" podStartSLOduration=1.8930638100000001 podStartE2EDuration="3.162208898s" totalImagesPullingTime="1.269145088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:58.161305523 +0000 UTC m=+624.976499948" watchObservedRunningTime="2026-10-30 13:01:58.162208898 +0000 UTC m=+624.977403321" Oct 30 13:01:58.428372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.428265 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v"] Oct 30 13:01:58.432156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.432121 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.448639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.448609 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 13:01:58.462516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.462483 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v"] Oct 30 13:01:58.575831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.575799 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:58.588533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.588504 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.588533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.588541 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.588779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.588573 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.588779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.588594 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.588779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.588726 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.588956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.588851 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8fgv\" (UniqueName: \"kubernetes.io/projected/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-kube-api-access-d8fgv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.680568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.680477 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:01:58.689915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.689887 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8fgv\" (UniqueName: \"kubernetes.io/projected/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-kube-api-access-d8fgv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.690408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.689960 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.690408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.690131 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.690408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.690189 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.690408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.690216 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.690408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.690248 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.690408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.690287 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.690703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.690529 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.690703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.690552 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.692891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.692865 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.693009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.692874 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.715744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.715707 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8fgv\" (UniqueName: \"kubernetes.io/projected/a53b4e96-a86d-4f7d-8fe9-727c0bb79b23-kube-api-access-d8fgv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lmr2v\" (UID: \"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:58.751451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:58.751413 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:01:59.059143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:59.059112 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v"] Oct 30 13:01:59.062027 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:01:59.061997 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53b4e96_a86d_4f7d_8fe9_727c0bb79b23.slice/crio-d58795e6e30c52fdf1a83ed01c6e4a42c2dd5c88538249641b96933d0718d071 WatchSource:0}: Error finding container d58795e6e30c52fdf1a83ed01c6e4a42c2dd5c88538249641b96933d0718d071: Status 404 returned error can't find the container with id d58795e6e30c52fdf1a83ed01c6e4a42c2dd5c88538249641b96933d0718d071 Oct 30 13:01:59.129627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:01:59.129590 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" event={"ID":"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23","Type":"ContainerStarted","Data":"d58795e6e30c52fdf1a83ed01c6e4a42c2dd5c88538249641b96933d0718d071"} Oct 30 13:02:00.134946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:00.134914 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" event={"ID":"a53b4e96-a86d-4f7d-8fe9-727c0bb79b23","Type":"ContainerStarted","Data":"a4d1dc2507ed727e31d2305009fbef8c610d475a971ab8cfe3e70188764d5885"} Oct 30 13:02:00.174765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:00.174709 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" podStartSLOduration=1.996221271 podStartE2EDuration="2.174695171s" totalImagesPullingTime="178.4739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:00.173817801 +0000 UTC m=+626.989012224" watchObservedRunningTime="2026-10-30 13:02:00.174695171 +0000 UTC m=+626.989889593" Oct 30 13:02:00.751768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:00.751730 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:02:00.898956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:00.898912 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:02.377467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.377436 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh"] Oct 30 13:02:02.380729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.380705 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.414339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.414289 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 13:02:02.419724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.419693 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh"] Oct 30 13:02:02.421004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.420976 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.421134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.421010 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.421134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.421040 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.421134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.421119 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqs5s\" (UniqueName: \"kubernetes.io/projected/579177ce-7692-4a1b-be47-09b365d67e25-kube-api-access-wqs5s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.421300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.421259 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.421379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.421308 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579177ce-7692-4a1b-be47-09b365d67e25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.522174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.522129 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.522377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.522181 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqs5s\" (UniqueName: \"kubernetes.io/projected/579177ce-7692-4a1b-be47-09b365d67e25-kube-api-access-wqs5s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.522377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.522221 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.522377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.522277 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579177ce-7692-4a1b-be47-09b365d67e25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.522377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.522344 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.522377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.522371 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.522726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.522701 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.522793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.522772 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.523041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.523011 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.525171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.525136 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579177ce-7692-4a1b-be47-09b365d67e25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.525515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.525495 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579177ce-7692-4a1b-be47-09b365d67e25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.569135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.569102 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:02.589398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.589360 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqs5s\" (UniqueName: \"kubernetes.io/projected/579177ce-7692-4a1b-be47-09b365d67e25-kube-api-access-wqs5s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh\" (UID: \"579177ce-7692-4a1b-be47-09b365d67e25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:02.694981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:02.694884 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:03.016278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:03.016250 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh"] Oct 30 13:02:03.018120 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:02:03.018080 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579177ce_7692_4a1b_be47_09b365d67e25.slice/crio-96377820891f16a66d26953b42377d81078a85cd038daf0a6e2ad1030b7f21fb WatchSource:0}: Error finding container 96377820891f16a66d26953b42377d81078a85cd038daf0a6e2ad1030b7f21fb: Status 404 returned error can't find the container with id 96377820891f16a66d26953b42377d81078a85cd038daf0a6e2ad1030b7f21fb Oct 30 13:02:03.150613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:03.150560 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" event={"ID":"579177ce-7692-4a1b-be47-09b365d67e25","Type":"ContainerStarted","Data":"96377820891f16a66d26953b42377d81078a85cd038daf0a6e2ad1030b7f21fb"} Oct 30 13:02:04.156984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:04.156948 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" event={"ID":"579177ce-7692-4a1b-be47-09b365d67e25","Type":"ContainerStarted","Data":"b4982f4393b3909684e55a9e6b247f1968d7467e0b32596648d5f8dd48db9951"} Oct 30 13:02:04.195959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:04.195908 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" podStartSLOduration=1.966020231 podStartE2EDuration="2.195893669s" totalImagesPullingTime="229.873438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:04.195285359 +0000 UTC m=+631.010479782" watchObservedRunningTime="2026-10-30 13:02:04.195893669 +0000 UTC m=+631.011088086" Oct 30 13:02:04.695244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:04.695208 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:05.680569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.680533 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:02:05.692995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.692968 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:02:05.723245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.723207 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6"] Oct 30 13:02:05.726899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.726868 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.741084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.741060 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 13:02:05.743438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.743413 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6"] Oct 30 13:02:05.759936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.759904 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfm9t\" (UniqueName: \"kubernetes.io/projected/dad6d45c-f49c-490c-906a-94b25d2b9430-kube-api-access-rfm9t\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.760070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.760019 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.760124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.760092 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad6d45c-f49c-490c-906a-94b25d2b9430-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.760172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.760152 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.760340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.760303 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.760406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.760350 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.830389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.830360 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:05.861584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.861546 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.861751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.861590 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.861751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.861650 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfm9t\" (UniqueName: \"kubernetes.io/projected/dad6d45c-f49c-490c-906a-94b25d2b9430-kube-api-access-rfm9t\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.861751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.861686 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.861751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.861712 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad6d45c-f49c-490c-906a-94b25d2b9430-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.861962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.861855 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.862031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.862006 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.862092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.862053 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.862291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.862261 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.864159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.864131 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad6d45c-f49c-490c-906a-94b25d2b9430-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.864291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.864273 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad6d45c-f49c-490c-906a-94b25d2b9430-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:05.887016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:05.886983 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfm9t\" (UniqueName: \"kubernetes.io/projected/dad6d45c-f49c-490c-906a-94b25d2b9430-kube-api-access-rfm9t\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6\" (UID: \"dad6d45c-f49c-490c-906a-94b25d2b9430\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:06.040607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:06.040526 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:06.180625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:06.180593 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk" Oct 30 13:02:06.379222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:06.379182 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6"] Oct 30 13:02:06.382245 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:02:06.382216 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddad6d45c_f49c_490c_906a_94b25d2b9430.slice/crio-6871277965e4d7ddd717612281f5e4d7cffbe36eee762a7f30bfd2ad7ad587e6 WatchSource:0}: Error finding container 6871277965e4d7ddd717612281f5e4d7cffbe36eee762a7f30bfd2ad7ad587e6: Status 404 returned error can't find the container with id 6871277965e4d7ddd717612281f5e4d7cffbe36eee762a7f30bfd2ad7ad587e6 Oct 30 13:02:07.174700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:07.174661 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" event={"ID":"dad6d45c-f49c-490c-906a-94b25d2b9430","Type":"ContainerStarted","Data":"9f272594ef6a92d08bf35385e1ad2f8ea8928444dd5ac3bbd5af703fa5cac773"} Oct 30 13:02:07.174700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:07.174708 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" event={"ID":"dad6d45c-f49c-490c-906a-94b25d2b9430","Type":"ContainerStarted","Data":"6871277965e4d7ddd717612281f5e4d7cffbe36eee762a7f30bfd2ad7ad587e6"} Oct 30 13:02:07.214816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:07.214745 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podStartSLOduration=2.027682135 podStartE2EDuration="2.21472837s" totalImagesPullingTime="187.046235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:07.213244551 +0000 UTC m=+634.028438974" watchObservedRunningTime="2026-10-30 13:02:07.21472837 +0000 UTC m=+634.029922794" Oct 30 13:02:07.581033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:07.580996 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:08.041593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:08.041506 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:08.752546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:08.752512 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:02:08.766255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:08.766228 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:02:09.199987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:09.199954 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lmr2v" Oct 30 13:02:12.695950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:12.695911 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:12.709158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:12.709134 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:13.223485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:13.223458 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh" Oct 30 13:02:16.041028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:16.040993 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:16.053267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:16.053241 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:16.238554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:16.238527 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:02:18.145659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.145616 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r"] Oct 30 13:02:18.152348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.151948 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.168150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.168128 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 13:02:18.176786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.176741 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bwksz\"" Oct 30 13:02:18.178718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.178695 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r"] Oct 30 13:02:18.278408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.278374 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.278584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.278483 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjm5\" (UniqueName: \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-kube-api-access-jqjm5\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.278584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.278513 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.278584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.278570 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.278721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.278594 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.278721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.278619 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.278721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.278646 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.278817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.278778 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.278817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.278803 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.379627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.379589 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.379627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.379634 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.379847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.379685 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.379847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.379756 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjm5\" (UniqueName: \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-kube-api-access-jqjm5\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.379847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.379784 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.379847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.379837 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.380029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.379872 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.380029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.379913 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.380029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.379948 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.380164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.380051 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.380164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.380135 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.380474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.380444 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.380608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.380534 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.380608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.380545 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.382288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.382263 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.382461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.382444 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.408074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.408004 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjm5\" (UniqueName: \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-kube-api-access-jqjm5\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.411989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.411970 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-kmj6r\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.465706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.465670 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:18.794976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:18.794942 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r"] Oct 30 13:02:18.798644 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:02:18.798618 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bc7804_9970_4bcf_90ed_fb7ef98b4471.slice/crio-1d40665a5680c5c93f640107705af89aedc6b0f0f0237235e091d843ea01581e WatchSource:0}: Error finding container 1d40665a5680c5c93f640107705af89aedc6b0f0f0237235e091d843ea01581e: Status 404 returned error can't find the container with id 1d40665a5680c5c93f640107705af89aedc6b0f0f0237235e091d843ea01581e Oct 30 13:02:19.243352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:19.243240 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" event={"ID":"48bc7804-9970-4bcf-90ed-fb7ef98b4471","Type":"ContainerStarted","Data":"1d40665a5680c5c93f640107705af89aedc6b0f0f0237235e091d843ea01581e"} Oct 30 13:02:21.603261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:21.603227 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:21.603671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:21.603308 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:21.603671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:21.603371 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:22.259600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:22.259562 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" event={"ID":"48bc7804-9970-4bcf-90ed-fb7ef98b4471","Type":"ContainerStarted","Data":"b80fc5b9991372b3f580d0b84c44a1c3cbecdad18ef9a0d921ffb46570486d1b"} Oct 30 13:02:22.331663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:22.331595 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" podStartSLOduration=1.529719693 podStartE2EDuration="4.33157509s" totalImagesPullingTime="2.801855397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:22.31783482 +0000 UTC m=+649.133029257" watchObservedRunningTime="2026-10-30 13:02:22.33157509 +0000 UTC m=+649.146769517" Oct 30 13:02:22.466018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:22.465979 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:22.467561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:22.467535 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-kmj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:02:22.467673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:22.467581 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 13:02:23.466256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:23.466218 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-kmj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:02:23.466761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:23.466277 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 13:02:24.432917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.432872 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:24.466902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.466871 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-kmj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:02:24.467262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.466934 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 13:02:24.578336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.578187 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-754459558f-vrhvp"] Oct 30 13:02:24.582331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.582304 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-754459558f-vrhvp" Oct 30 13:02:24.628175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.628139 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-754459558f-vrhvp"] Oct 30 13:02:24.754580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.754499 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkf75\" (UniqueName: \"kubernetes.io/projected/152af948-ed7c-44ff-b10f-40bc2030b17f-kube-api-access-dkf75\") pod \"authorino-754459558f-vrhvp\" (UID: \"152af948-ed7c-44ff-b10f-40bc2030b17f\") " pod="kuadrant-system/authorino-754459558f-vrhvp" Oct 30 13:02:24.754735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.754632 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/152af948-ed7c-44ff-b10f-40bc2030b17f-tls-cert\") pod \"authorino-754459558f-vrhvp\" (UID: \"152af948-ed7c-44ff-b10f-40bc2030b17f\") " pod="kuadrant-system/authorino-754459558f-vrhvp" Oct 30 13:02:24.855194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.855147 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/152af948-ed7c-44ff-b10f-40bc2030b17f-tls-cert\") pod \"authorino-754459558f-vrhvp\" (UID: \"152af948-ed7c-44ff-b10f-40bc2030b17f\") " pod="kuadrant-system/authorino-754459558f-vrhvp" Oct 30 13:02:24.855404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.855232 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkf75\" (UniqueName: \"kubernetes.io/projected/152af948-ed7c-44ff-b10f-40bc2030b17f-kube-api-access-dkf75\") pod \"authorino-754459558f-vrhvp\" (UID: \"152af948-ed7c-44ff-b10f-40bc2030b17f\") " pod="kuadrant-system/authorino-754459558f-vrhvp" Oct 30 13:02:24.857975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.857950 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/152af948-ed7c-44ff-b10f-40bc2030b17f-tls-cert\") pod \"authorino-754459558f-vrhvp\" (UID: \"152af948-ed7c-44ff-b10f-40bc2030b17f\") " pod="kuadrant-system/authorino-754459558f-vrhvp" Oct 30 13:02:24.883639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.883613 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkf75\" (UniqueName: \"kubernetes.io/projected/152af948-ed7c-44ff-b10f-40bc2030b17f-kube-api-access-dkf75\") pod \"authorino-754459558f-vrhvp\" (UID: \"152af948-ed7c-44ff-b10f-40bc2030b17f\") " pod="kuadrant-system/authorino-754459558f-vrhvp" Oct 30 13:02:24.893755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:24.893727 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-754459558f-vrhvp" Oct 30 13:02:25.233127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:25.233087 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-754459558f-vrhvp"] Oct 30 13:02:25.238310 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:02:25.238268 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod152af948_ed7c_44ff_b10f_40bc2030b17f.slice/crio-4494b76713374eb89f62484fd835a72fc6333573fce1ef7e8ab42d0af9de3e02 WatchSource:0}: Error finding container 4494b76713374eb89f62484fd835a72fc6333573fce1ef7e8ab42d0af9de3e02: Status 404 returned error can't find the container with id 4494b76713374eb89f62484fd835a72fc6333573fce1ef7e8ab42d0af9de3e02 Oct 30 13:02:25.256566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:25.256547 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:25.274763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:25.274716 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-754459558f-vrhvp" event={"ID":"152af948-ed7c-44ff-b10f-40bc2030b17f","Type":"ContainerStarted","Data":"4494b76713374eb89f62484fd835a72fc6333573fce1ef7e8ab42d0af9de3e02"} Oct 30 13:02:25.466995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:25.466958 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-kmj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:02:25.467414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:25.467013 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 13:02:25.589141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:25.589109 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:26.280686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.280589 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-754459558f-vrhvp" event={"ID":"152af948-ed7c-44ff-b10f-40bc2030b17f","Type":"ContainerStarted","Data":"4fa9cd67a6c789e499e1b917da7a03ff325cc548a63b1a771a2d58bcd98bfdd3"} Oct 30 13:02:26.331447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.330578 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-754459558f-vrhvp" podStartSLOduration=1.568504292 podStartE2EDuration="2.330556808s" totalImagesPullingTime="762.052516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:26.330458306 +0000 UTC m=+653.145652729" watchObservedRunningTime="2026-10-30 13:02:26.330556808 +0000 UTC m=+653.145751232" Oct 30 13:02:26.433729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.433698 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-86587766db-n2bh7"] Oct 30 13:02:26.434093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.434072 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-86587766db-n2bh7" podUID="29b1f0bb-60a8-45e8-910b-284834a48f47" containerName="authorino" containerID="cri-o://cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348" gracePeriod=30 Oct 30 13:02:26.467167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.467142 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-kmj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:02:26.467586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.467192 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 13:02:26.690077 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:02:26.690034 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:02:26.701083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.701055 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86587766db-n2bh7" Oct 30 13:02:26.874794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.874711 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b1f0bb-60a8-45e8-910b-284834a48f47-kube-api-access-shbw2\" (UniqueName: \"kubernetes.io/projected/29b1f0bb-60a8-45e8-910b-284834a48f47-kube-api-access-shbw2\") pod \"29b1f0bb-60a8-45e8-910b-284834a48f47\" (UID: \"29b1f0bb-60a8-45e8-910b-284834a48f47\") " Oct 30 13:02:26.874794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.874777 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29b1f0bb-60a8-45e8-910b-284834a48f47-tls-cert\" (UniqueName: \"kubernetes.io/secret/29b1f0bb-60a8-45e8-910b-284834a48f47-tls-cert\") pod \"29b1f0bb-60a8-45e8-910b-284834a48f47\" (UID: \"29b1f0bb-60a8-45e8-910b-284834a48f47\") " Oct 30 13:02:26.877475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.877439 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b1f0bb-60a8-45e8-910b-284834a48f47-kube-api-access-shbw2" pod "29b1f0bb-60a8-45e8-910b-284834a48f47" (UID: "29b1f0bb-60a8-45e8-910b-284834a48f47"). InnerVolumeSpecName "kube-api-access-shbw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:26.886828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.886801 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b1f0bb-60a8-45e8-910b-284834a48f47-tls-cert" pod "29b1f0bb-60a8-45e8-910b-284834a48f47" (UID: "29b1f0bb-60a8-45e8-910b-284834a48f47"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:26.976014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.975984 5049 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29b1f0bb-60a8-45e8-910b-284834a48f47-tls-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:02:26.976014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:26.976011 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shbw2\" (UniqueName: \"kubernetes.io/projected/29b1f0bb-60a8-45e8-910b-284834a48f47-kube-api-access-shbw2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:02:27.290392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.289702 5049 generic.go:356] "Generic (PLEG): container finished" podID="29b1f0bb-60a8-45e8-910b-284834a48f47" containerID="cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348" exitCode=0 Oct 30 13:02:27.290727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.290694 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86587766db-n2bh7" event={"ID":"29b1f0bb-60a8-45e8-910b-284834a48f47","Type":"ContainerDied","Data":"cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348"} Oct 30 13:02:27.293730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.290798 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86587766db-n2bh7" Oct 30 13:02:27.294036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.293703 5049 scope.go:122] "RemoveContainer" containerID="cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348" Oct 30 13:02:27.294183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.293710 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86587766db-n2bh7" event={"ID":"29b1f0bb-60a8-45e8-910b-284834a48f47","Type":"ContainerDied","Data":"b143abf1df5349152fb3c5746d7011518318e7566d65b0fecde8e042365689c4"} Oct 30 13:02:27.330504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.325769 5049 scope.go:122] "RemoveContainer" containerID="cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348" Oct 30 13:02:27.330504 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:02:27.326156 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348\": container with ID starting with cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348 not found: ID does not exist" containerID="cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348" Oct 30 13:02:27.330504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.326196 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348"} err="failed to get container status \"cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348\": rpc error: code = NotFound desc = could not find container \"cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348\": container with ID starting with cd30e748f8d1ecb2361f9f4473f47dbf477ba3a58bcc2ae3187a4a64bc307348 not found: ID does not exist" Oct 30 13:02:27.364105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.360813 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-86587766db-n2bh7"] Oct 30 13:02:27.375144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.374824 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-86587766db-n2bh7"] Oct 30 13:02:27.467019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.466987 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-kmj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:02:27.467187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:27.467044 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 13:02:28.466473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:28.466437 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-kmj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:02:28.466672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:28.466501 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 13:02:29.341343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.340708 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b1f0bb-60a8-45e8-910b-284834a48f47" path="/var/lib/kubelet/pods/29b1f0bb-60a8-45e8-910b-284834a48f47/volumes" Oct 30 13:02:29.646590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.646501 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:29.782770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.782726 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm"] Oct 30 13:02:29.783120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.783105 5049 cpu_manager.go:414] "containerMap: removing container" podUID="29b1f0bb-60a8-45e8-910b-284834a48f47" containerName="authorino" Oct 30 13:02:29.783120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.783118 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b1f0bb-60a8-45e8-910b-284834a48f47" containerName="authorino" Oct 30 13:02:29.783272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.783193 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b1f0bb-60a8-45e8-910b-284834a48f47" containerName="authorino" Oct 30 13:02:29.788461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.788440 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" Oct 30 13:02:29.817340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.817289 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 13:02:29.827903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.827844 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm"] Oct 30 13:02:29.910142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.910035 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ace317-daee-47d5-a744-10547abad774-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9mvqm\" (UID: \"a1ace317-daee-47d5-a744-10547abad774\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" Oct 30 13:02:29.910142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:29.910105 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r52g\" (UniqueName: \"kubernetes.io/projected/a1ace317-daee-47d5-a744-10547abad774-kube-api-access-4r52g\") pod \"payload-pre-processing-7b98bb678c-9mvqm\" (UID: \"a1ace317-daee-47d5-a744-10547abad774\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" Oct 30 13:02:30.011595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:30.011556 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ace317-daee-47d5-a744-10547abad774-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9mvqm\" (UID: \"a1ace317-daee-47d5-a744-10547abad774\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" Oct 30 13:02:30.011878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:30.011849 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r52g\" (UniqueName: \"kubernetes.io/projected/a1ace317-daee-47d5-a744-10547abad774-kube-api-access-4r52g\") pod \"payload-pre-processing-7b98bb678c-9mvqm\" (UID: \"a1ace317-daee-47d5-a744-10547abad774\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" Oct 30 13:02:30.012643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:30.012606 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ace317-daee-47d5-a744-10547abad774-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9mvqm\" (UID: \"a1ace317-daee-47d5-a744-10547abad774\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" Oct 30 13:02:30.046676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:30.042468 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:30.046676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:30.046045 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r52g\" (UniqueName: \"kubernetes.io/projected/a1ace317-daee-47d5-a744-10547abad774-kube-api-access-4r52g\") pod \"payload-pre-processing-7b98bb678c-9mvqm\" (UID: \"a1ace317-daee-47d5-a744-10547abad774\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" Oct 30 13:02:30.103501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:30.103464 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" Oct 30 13:02:30.174537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:30.173199 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:30.174537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:30.174342 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:30.175878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:30.175814 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:02:30.456159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:30.456069 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm"] Oct 30 13:02:30.460270 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:02:30.460230 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ace317_daee_47d5_a744_10547abad774.slice/crio-a9e0c6a1169fbb5e0f6d6e73c6abaca4359e3d80f4a8cabed1a2c5c2377acd5f WatchSource:0}: Error finding container a9e0c6a1169fbb5e0f6d6e73c6abaca4359e3d80f4a8cabed1a2c5c2377acd5f: Status 404 returned error can't find the container with id a9e0c6a1169fbb5e0f6d6e73c6abaca4359e3d80f4a8cabed1a2c5c2377acd5f Oct 30 13:02:31.327651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:31.327615 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" event={"ID":"a1ace317-daee-47d5-a744-10547abad774","Type":"ContainerStarted","Data":"a9e0c6a1169fbb5e0f6d6e73c6abaca4359e3d80f4a8cabed1a2c5c2377acd5f"} Oct 30 13:02:33.340700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:33.340664 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" event={"ID":"a1ace317-daee-47d5-a744-10547abad774","Type":"ContainerStarted","Data":"6d3321c79e165f85be4a7d8c11bd27f1ef9b8662d2ba23422d0861a9e2bf8cee"} Oct 30 13:02:33.341107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:33.340897 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" Oct 30 13:02:33.378979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:33.378924 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" podStartSLOduration=2.009346468 podStartE2EDuration="4.37891061s" totalImagesPullingTime="2.369564142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:33.378476663 +0000 UTC m=+660.193671086" watchObservedRunningTime="2026-10-30 13:02:33.37891061 +0000 UTC m=+660.194105033" Oct 30 13:02:41.284403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.284363 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-d5f86698f-jvzrj"] Oct 30 13:02:41.289504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.289482 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" Oct 30 13:02:41.303285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.303265 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-bkjx4\"" Oct 30 13:02:41.313827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.313805 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 13:02:41.313926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.313815 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 13:02:41.315863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.315836 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d5f86698f-jvzrj"] Oct 30 13:02:41.426046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.426010 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b3822a7-c018-4112-a352-f436ce4b2de9-metrics-certs\") pod \"ai-gateway-controller-d5f86698f-jvzrj\" (UID: \"7b3822a7-c018-4112-a352-f436ce4b2de9\") " pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" Oct 30 13:02:41.426211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.426150 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7879\" (UniqueName: \"kubernetes.io/projected/7b3822a7-c018-4112-a352-f436ce4b2de9-kube-api-access-j7879\") pod \"ai-gateway-controller-d5f86698f-jvzrj\" (UID: \"7b3822a7-c018-4112-a352-f436ce4b2de9\") " pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" Oct 30 13:02:41.526957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.526921 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b3822a7-c018-4112-a352-f436ce4b2de9-metrics-certs\") pod \"ai-gateway-controller-d5f86698f-jvzrj\" (UID: \"7b3822a7-c018-4112-a352-f436ce4b2de9\") " pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" Oct 30 13:02:41.527105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.527001 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7879\" (UniqueName: \"kubernetes.io/projected/7b3822a7-c018-4112-a352-f436ce4b2de9-kube-api-access-j7879\") pod \"ai-gateway-controller-d5f86698f-jvzrj\" (UID: \"7b3822a7-c018-4112-a352-f436ce4b2de9\") " pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" Oct 30 13:02:41.529427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.529397 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b3822a7-c018-4112-a352-f436ce4b2de9-metrics-certs\") pod \"ai-gateway-controller-d5f86698f-jvzrj\" (UID: \"7b3822a7-c018-4112-a352-f436ce4b2de9\") " pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" Oct 30 13:02:41.548428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.548405 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7879\" (UniqueName: \"kubernetes.io/projected/7b3822a7-c018-4112-a352-f436ce4b2de9-kube-api-access-j7879\") pod \"ai-gateway-controller-d5f86698f-jvzrj\" (UID: \"7b3822a7-c018-4112-a352-f436ce4b2de9\") " pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" Oct 30 13:02:41.602410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.602384 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" Oct 30 13:02:41.898277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:41.898245 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d5f86698f-jvzrj"] Oct 30 13:02:42.381690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:42.381654 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" event={"ID":"7b3822a7-c018-4112-a352-f436ce4b2de9","Type":"ContainerStarted","Data":"10f15ed2a119ef67d3c22b5615c5aa2b336efda89104a039fa03d956cbab740a"} Oct 30 13:02:44.348393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:44.348362 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9mvqm" Oct 30 13:02:44.397535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:44.397492 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" event={"ID":"7b3822a7-c018-4112-a352-f436ce4b2de9","Type":"ContainerStarted","Data":"2b4b008439c87169e9b17e5baf5dd214131e79ea43d3b84efe1fea5e575d97ca"} Oct 30 13:02:44.397731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:44.397572 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" Oct 30 13:02:44.436985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:44.436922 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" podStartSLOduration=1.132408459 podStartE2EDuration="3.436906903s" totalImagesPullingTime="2.304498444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:44.4347184 +0000 UTC m=+671.249912823" watchObservedRunningTime="2026-10-30 13:02:44.436906903 +0000 UTC m=+671.252101329" Oct 30 13:02:50.590530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:50.590493 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:55.403939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:55.403909 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-d5f86698f-jvzrj" Oct 30 13:02:56.675532 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:02:56.675495 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:02:57.166905 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:57.166870 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:57.660664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:57.660636 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:59.286124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:59.286090 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:59.469230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:02:59.469200 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:00.859391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:00.859359 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:01.018782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:01.018750 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:21.372504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.372472 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:21.529218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.529184 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:21.783507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.783424 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8"] Oct 30 13:03:21.787720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.787690 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.813729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.813704 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-52a649-openshift-default-dockercfg-frxr8\"" Oct 30 13:03:21.827520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.827493 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8"] Oct 30 13:03:21.880574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.880536 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7m7\" (UniqueName: \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-kube-api-access-gp7m7\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.880743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.880589 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.880743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.880635 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.880743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.880675 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.880743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.880714 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.880743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.880730 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.881035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.880847 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.881035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.880877 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.881035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.880902 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.887709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.887685 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:21.982361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982310 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.982528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982409 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.982528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982442 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.982528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982495 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.982528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982520 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.982730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982595 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.982730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982634 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.982730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982666 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.982967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982738 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.982967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982745 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7m7\" (UniqueName: \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-kube-api-access-gp7m7\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.983070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.982999 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.983541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.983412 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.983793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.983769 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.983999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.983978 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.985360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.985338 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:21.985590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:21.985568 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:22.006623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.006601 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:22.006789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.006770 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7m7\" (UniqueName: \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-kube-api-access-gp7m7\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:22.079836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.079797 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:22.080920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.080895 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb"] Oct 30 13:03:22.086165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.086137 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.102277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.102244 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d7e62b-openshift-default-dockercfg-29w5v\"" Oct 30 13:03:22.102646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.102623 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:22.116594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.116567 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb"] Oct 30 13:03:22.190113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.190017 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.190113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.190070 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.190113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.190100 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.190957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.190128 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.190957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.190184 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.190957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.190212 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7kc\" (UniqueName: \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-kube-api-access-vn7kc\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.190957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.190282 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df030fd-edbd-4631-b7e5-292564e90d08-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.190957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.190513 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.191214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.191131 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df030fd-edbd-4631-b7e5-292564e90d08-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.291858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.291826 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.291880 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.291904 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7kc\" (UniqueName: \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-kube-api-access-vn7kc\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.291969 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df030fd-edbd-4631-b7e5-292564e90d08-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.292006 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.292029 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df030fd-edbd-4631-b7e5-292564e90d08-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.292098 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.292137 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.292166 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.292302 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.292358 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.292655 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df030fd-edbd-4631-b7e5-292564e90d08-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.292767 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.292836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.292813 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.294652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.294631 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.294729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.294716 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df030fd-edbd-4631-b7e5-292564e90d08-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.332929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.332868 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.334247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.334226 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7kc\" (UniqueName: \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-kube-api-access-vn7kc\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.381783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.381743 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:22.403945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.403912 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:22.453569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.453524 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m"] Oct 30 13:03:22.460130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.458784 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8"] Oct 30 13:03:22.460130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.458986 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.462929 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:03:22.462903 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ad6423_a145_4a83_be4f_12fad5cb0a51.slice/crio-3c115e23f5319f14efc9a1ce02f78e4211e7d60757e6d9ec7a84b43ead9b1680 WatchSource:0}: Error finding container 3c115e23f5319f14efc9a1ce02f78e4211e7d60757e6d9ec7a84b43ead9b1680: Status 404 returned error can't find the container with id 3c115e23f5319f14efc9a1ce02f78e4211e7d60757e6d9ec7a84b43ead9b1680 Oct 30 13:03:22.466114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.466065 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:22.466181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.466147 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:22.466237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.466197 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:22.480982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.480158 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f07dfe-openshift-default-dockercfg-d79w9\"" Oct 30 13:03:22.484422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.484358 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m"] Oct 30 13:03:22.558300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.558261 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:22.585881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.585782 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" event={"ID":"c7ad6423-a145-4a83-be4f-12fad5cb0a51","Type":"ContainerStarted","Data":"1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442"} Oct 30 13:03:22.585881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.585845 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" event={"ID":"c7ad6423-a145-4a83-be4f-12fad5cb0a51","Type":"ContainerStarted","Data":"3c115e23f5319f14efc9a1ce02f78e4211e7d60757e6d9ec7a84b43ead9b1680"} Oct 30 13:03:22.595858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.595819 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87855fb7-df0f-468b-9b00-55f680e3a242-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.596124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.596099 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.596229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.596171 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.596229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.596213 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.596358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.596269 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbcmg\" (UniqueName: \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-kube-api-access-xbcmg\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.596358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.596310 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.596472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.596414 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87855fb7-df0f-468b-9b00-55f680e3a242-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.596472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.596439 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.596576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.596481 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.641529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.641465 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" podStartSLOduration=1.641443188 podStartE2EDuration="1.641443188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:22.638699677 +0000 UTC m=+709.453894102" watchObservedRunningTime="2026-10-30 13:03:22.641443188 +0000 UTC m=+709.456637611" Oct 30 13:03:22.697062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697015 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbcmg\" (UniqueName: \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-kube-api-access-xbcmg\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697096 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697166 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87855fb7-df0f-468b-9b00-55f680e3a242-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697193 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697293 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697356 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87855fb7-df0f-468b-9b00-55f680e3a242-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697384 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697458 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697497 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697714 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697768 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.697961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.697943 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.698443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.698192 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.698614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.698579 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87855fb7-df0f-468b-9b00-55f680e3a242-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.700624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.700567 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.700733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.700714 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87855fb7-df0f-468b-9b00-55f680e3a242-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.723034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.723006 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb"] Oct 30 13:03:22.723287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.723264 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbcmg\" (UniqueName: \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-kube-api-access-xbcmg\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.723714 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:03:22.723684 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df030fd_edbd_4631_b7e5_292564e90d08.slice/crio-ff144ee82740e6e7575a8750f294ef3d3d80d715e5b90872d35e0d0bda1cb997 WatchSource:0}: Error finding container ff144ee82740e6e7575a8750f294ef3d3d80d715e5b90872d35e0d0bda1cb997: Status 404 returned error can't find the container with id ff144ee82740e6e7575a8750f294ef3d3d80d715e5b90872d35e0d0bda1cb997 Oct 30 13:03:22.726107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.726077 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:22.726199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.726132 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:22.726199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.726162 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:22.727012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.726990 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.734774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.734750 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:22.759376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.753793 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb"] Oct 30 13:03:22.787362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.785282 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:22.793486 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.792556 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:22.800582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.798543 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb"] Oct 30 13:03:22.802194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.802169 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d833bd-openshift-default-dockercfg-dpcql\"" Oct 30 13:03:22.899000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.898966 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:22.899269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.899247 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:22.899490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.899454 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpqfw\" (UniqueName: \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-kube-api-access-qpqfw\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:22.899692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.899658 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:22.899856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.899835 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:22.900038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.900024 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:22.900241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.900226 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:22.900438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.900404 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:22.900633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.900619 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:22.902136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:22.902112 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:23.001449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.001405 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.001638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.001456 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.001638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.001504 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.001638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.001565 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.001638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.001596 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.001638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.001619 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpqfw\" (UniqueName: \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-kube-api-access-qpqfw\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.001902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.001827 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.001902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.001857 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.001997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.001917 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.002362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.002316 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.003048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.002759 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.003048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.003000 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.003380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.003359 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.003467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.003392 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.005445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.005407 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.005634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.005615 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.028138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.028108 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.028280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.028225 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpqfw\" (UniqueName: \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-kube-api-access-qpqfw\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.063827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.063796 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:23.103195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.103155 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:23.108248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.108220 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:23.121563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.121501 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:23.130549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.130425 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m"] Oct 30 13:03:23.138379 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:03:23.138352 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87855fb7_df0f_468b_9b00_55f680e3a242.slice/crio-8cb06e0970f886d97059559171ac1bafdb4c137f2069ca41e6e1e732cf6c417b WatchSource:0}: Error finding container 8cb06e0970f886d97059559171ac1bafdb4c137f2069ca41e6e1e732cf6c417b: Status 404 returned error can't find the container with id 8cb06e0970f886d97059559171ac1bafdb4c137f2069ca41e6e1e732cf6c417b Oct 30 13:03:23.141567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.141533 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:23.141656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.141618 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:23.141767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.141659 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:23.439734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.439704 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb"] Oct 30 13:03:23.440729 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:03:23.440702 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e20740_a55b_4ca5_8f06_f01b8b9cd57d.slice/crio-e4b08914b1904de33fa1bf771cb65f650c8971fb7ebceebf16f38148f003ac81 WatchSource:0}: Error finding container e4b08914b1904de33fa1bf771cb65f650c8971fb7ebceebf16f38148f003ac81: Status 404 returned error can't find the container with id e4b08914b1904de33fa1bf771cb65f650c8971fb7ebceebf16f38148f003ac81 Oct 30 13:03:23.443257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.443227 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:23.443395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.443291 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:23.443395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.443347 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:23.593474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.593435 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" event={"ID":"06e20740-a55b-4ca5-8f06-f01b8b9cd57d","Type":"ContainerStarted","Data":"5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041"} Oct 30 13:03:23.593679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.593492 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" event={"ID":"06e20740-a55b-4ca5-8f06-f01b8b9cd57d","Type":"ContainerStarted","Data":"e4b08914b1904de33fa1bf771cb65f650c8971fb7ebceebf16f38148f003ac81"} Oct 30 13:03:23.595771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.595738 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" event={"ID":"9df030fd-edbd-4631-b7e5-292564e90d08","Type":"ContainerStarted","Data":"73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d"} Oct 30 13:03:23.595913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.595793 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" event={"ID":"9df030fd-edbd-4631-b7e5-292564e90d08","Type":"ContainerStarted","Data":"ff144ee82740e6e7575a8750f294ef3d3d80d715e5b90872d35e0d0bda1cb997"} Oct 30 13:03:23.598130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.598101 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" event={"ID":"87855fb7-df0f-468b-9b00-55f680e3a242","Type":"ContainerStarted","Data":"17a2c00c77578ef6aea3e7dcb338b55cfabf690e3bb4ecea302b29b97a116958"} Oct 30 13:03:23.598240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.598145 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" event={"ID":"87855fb7-df0f-468b-9b00-55f680e3a242","Type":"ContainerStarted","Data":"8cb06e0970f886d97059559171ac1bafdb4c137f2069ca41e6e1e732cf6c417b"} Oct 30 13:03:23.598533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.598513 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:23.599611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.599593 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:23.639808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.639744 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" podStartSLOduration=1.6397273079999999 podStartE2EDuration="1.639727308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:23.638531648 +0000 UTC m=+710.453726074" watchObservedRunningTime="2026-10-30 13:03:23.639727308 +0000 UTC m=+710.454921731" Oct 30 13:03:23.690703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.690596 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" podStartSLOduration=1.690582695 podStartE2EDuration="1.690582695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:23.68856112 +0000 UTC m=+710.503755544" watchObservedRunningTime="2026-10-30 13:03:23.690582695 +0000 UTC m=+710.505777117" Oct 30 13:03:23.736546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.736486 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" podStartSLOduration=1.736467197 podStartE2EDuration="1.736467197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:23.733740284 +0000 UTC m=+710.548934709" watchObservedRunningTime="2026-10-30 13:03:23.736467197 +0000 UTC m=+710.551661621" Oct 30 13:03:23.793338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:23.793277 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:24.122500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:24.122467 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:24.127614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:24.127589 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:24.404453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:24.404371 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:24.409590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:24.409565 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:24.532924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:24.532874 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:24.603608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:24.603578 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:24.603767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:24.603631 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:24.604196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:24.604172 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:24.604347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:24.604303 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:03:24.798519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:24.798488 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:25.400657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:25.400621 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:25.608633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:25.608600 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:25.609154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:25.609137 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:25.642114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:25.642089 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:25.951940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:25.951896 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.339257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:26.339206 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.572261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:26.572230 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.676673 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:03:26.676591 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:03:27.181986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.181930 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.336295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.336265 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm"] Oct 30 13:03:27.340905 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.340881 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:03:27.381944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.381908 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm"] Oct 30 13:03:27.385187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.385162 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-52a649\"" Oct 30 13:03:27.394703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.394670 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-52a649-dockercfg-dnbdp\"" Oct 30 13:03:27.438995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.438921 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.450229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.450200 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7tzn\" (UniqueName: \"kubernetes.io/projected/3e86a92d-8512-45b0-8880-ef6432fcf7ff-kube-api-access-g7tzn\") pod \"payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm\" (UID: \"3e86a92d-8512-45b0-8880-ef6432fcf7ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:03:27.450390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.450262 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e86a92d-8512-45b0-8880-ef6432fcf7ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm\" (UID: \"3e86a92d-8512-45b0-8880-ef6432fcf7ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:03:27.551213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.551180 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7tzn\" (UniqueName: \"kubernetes.io/projected/3e86a92d-8512-45b0-8880-ef6432fcf7ff-kube-api-access-g7tzn\") pod \"payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm\" (UID: \"3e86a92d-8512-45b0-8880-ef6432fcf7ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:03:27.551418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.551241 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e86a92d-8512-45b0-8880-ef6432fcf7ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm\" (UID: \"3e86a92d-8512-45b0-8880-ef6432fcf7ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:03:27.553446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.552009 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e86a92d-8512-45b0-8880-ef6432fcf7ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm\" (UID: \"3e86a92d-8512-45b0-8880-ef6432fcf7ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:03:27.575854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.575818 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7tzn\" (UniqueName: \"kubernetes.io/projected/3e86a92d-8512-45b0-8880-ef6432fcf7ff-kube-api-access-g7tzn\") pod \"payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm\" (UID: \"3e86a92d-8512-45b0-8880-ef6432fcf7ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:03:27.636809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.636772 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.655331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.655283 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:03:27.798013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:27.796369 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:28.010422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:28.010394 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm"] Oct 30 13:03:28.011070 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:03:28.011037 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e86a92d_8512_45b0_8880_ef6432fcf7ff.slice/crio-547ada52589864010de8203e005dd9318e2214d49c3a2b6638daae428aacfe0b WatchSource:0}: Error finding container 547ada52589864010de8203e005dd9318e2214d49c3a2b6638daae428aacfe0b: Status 404 returned error can't find the container with id 547ada52589864010de8203e005dd9318e2214d49c3a2b6638daae428aacfe0b Oct 30 13:03:28.030660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:28.030639 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:28.276040 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:28.275939 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:28.521845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:28.521814 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:28.624966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:28.624923 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" event={"ID":"3e86a92d-8512-45b0-8880-ef6432fcf7ff","Type":"ContainerStarted","Data":"05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5"} Oct 30 13:03:28.624966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:28.624974 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" event={"ID":"3e86a92d-8512-45b0-8880-ef6432fcf7ff","Type":"ContainerStarted","Data":"547ada52589864010de8203e005dd9318e2214d49c3a2b6638daae428aacfe0b"} Oct 30 13:03:28.625463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:28.625022 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:03:28.690897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:28.690844 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" podStartSLOduration=1.690829735 podStartE2EDuration="1.690829735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:28.690707726 +0000 UTC m=+715.505902150" watchObservedRunningTime="2026-10-30 13:03:28.690829735 +0000 UTC m=+715.506024182" Oct 30 13:03:29.177018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:29.176985 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:29.360772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:29.360733 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:30.581076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:30.581045 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:31.655999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:31.655965 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:31.831342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:31.830788 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:32.520498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.520466 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:32.577627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.577576 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2"] Oct 30 13:03:32.596356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.586486 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:03:32.598374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.598337 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede810a9-3a02-490d-9a25-d0788bea4cdf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2\" (UID: \"ede810a9-3a02-490d-9a25-d0788bea4cdf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:03:32.598829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.598804 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbl96\" (UniqueName: \"kubernetes.io/projected/ede810a9-3a02-490d-9a25-d0788bea4cdf-kube-api-access-fbl96\") pod \"payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2\" (UID: \"ede810a9-3a02-490d-9a25-d0788bea4cdf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:03:32.623249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.613583 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f07dfe-dockercfg-bzhjg\"" Oct 30 13:03:32.623249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.614807 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f07dfe\"" Oct 30 13:03:32.641385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.637315 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj"] Oct 30 13:03:32.665588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.665558 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2"] Oct 30 13:03:32.666140 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.666121 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj"] Oct 30 13:03:32.666275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.665991 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:03:32.700667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.700634 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede810a9-3a02-490d-9a25-d0788bea4cdf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2\" (UID: \"ede810a9-3a02-490d-9a25-d0788bea4cdf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:03:32.700862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.700682 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409baa7e-092d-4d07-9a46-e611890a8458-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj\" (UID: \"409baa7e-092d-4d07-9a46-e611890a8458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:03:32.700862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.700729 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msp8s\" (UniqueName: \"kubernetes.io/projected/409baa7e-092d-4d07-9a46-e611890a8458-kube-api-access-msp8s\") pod \"payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj\" (UID: \"409baa7e-092d-4d07-9a46-e611890a8458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:03:32.700862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.700764 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbl96\" (UniqueName: \"kubernetes.io/projected/ede810a9-3a02-490d-9a25-d0788bea4cdf-kube-api-access-fbl96\") pod \"payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2\" (UID: \"ede810a9-3a02-490d-9a25-d0788bea4cdf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:03:32.701542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.701512 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede810a9-3a02-490d-9a25-d0788bea4cdf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2\" (UID: \"ede810a9-3a02-490d-9a25-d0788bea4cdf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:03:32.730264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.730200 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbl96\" (UniqueName: \"kubernetes.io/projected/ede810a9-3a02-490d-9a25-d0788bea4cdf-kube-api-access-fbl96\") pod \"payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2\" (UID: \"ede810a9-3a02-490d-9a25-d0788bea4cdf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:03:32.746722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.746591 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:32.802393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.802283 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msp8s\" (UniqueName: \"kubernetes.io/projected/409baa7e-092d-4d07-9a46-e611890a8458-kube-api-access-msp8s\") pod \"payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj\" (UID: \"409baa7e-092d-4d07-9a46-e611890a8458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:03:32.802848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.802780 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409baa7e-092d-4d07-9a46-e611890a8458-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj\" (UID: \"409baa7e-092d-4d07-9a46-e611890a8458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:03:32.803582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.803561 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409baa7e-092d-4d07-9a46-e611890a8458-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj\" (UID: \"409baa7e-092d-4d07-9a46-e611890a8458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:03:32.826818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.826792 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msp8s\" (UniqueName: \"kubernetes.io/projected/409baa7e-092d-4d07-9a46-e611890a8458-kube-api-access-msp8s\") pod \"payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj\" (UID: \"409baa7e-092d-4d07-9a46-e611890a8458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:03:32.905601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.905565 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:32.920500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.920466 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:03:32.986897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:32.986617 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:03:33.056237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.056171 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:33.243828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.243794 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2"] Oct 30 13:03:33.245840 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:03:33.245816 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede810a9_3a02_490d_9a25_d0788bea4cdf.slice/crio-7be712c9eca1ad7027e6897e7e3d6cc89eca570cd6e4436ffb687062a9ff1493 WatchSource:0}: Error finding container 7be712c9eca1ad7027e6897e7e3d6cc89eca570cd6e4436ffb687062a9ff1493: Status 404 returned error can't find the container with id 7be712c9eca1ad7027e6897e7e3d6cc89eca570cd6e4436ffb687062a9ff1493 Oct 30 13:03:33.373845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.373684 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj"] Oct 30 13:03:33.679560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.679442 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" event={"ID":"409baa7e-092d-4d07-9a46-e611890a8458","Type":"ContainerStarted","Data":"2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050"} Oct 30 13:03:33.679560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.679525 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:03:33.680061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.679564 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" event={"ID":"409baa7e-092d-4d07-9a46-e611890a8458","Type":"ContainerStarted","Data":"5d3ffd995a03b348f35e924e64e7bfe4fac230be5f7865f5d2ebf45971f4418b"} Oct 30 13:03:33.681126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.681096 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" event={"ID":"ede810a9-3a02-490d-9a25-d0788bea4cdf","Type":"ContainerStarted","Data":"344153ca7d7bd1782a71b1b1370fcd1ace0fe26860056380a2de00b4726f5113"} Oct 30 13:03:33.681261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.681144 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" event={"ID":"ede810a9-3a02-490d-9a25-d0788bea4cdf","Type":"ContainerStarted","Data":"7be712c9eca1ad7027e6897e7e3d6cc89eca570cd6e4436ffb687062a9ff1493"} Oct 30 13:03:33.681261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.681191 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:03:33.727675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.727190 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" podStartSLOduration=1.727170658 podStartE2EDuration="1.727170658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:33.725619288 +0000 UTC m=+720.540813724" watchObservedRunningTime="2026-10-30 13:03:33.727170658 +0000 UTC m=+720.542365082" Oct 30 13:03:33.768268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:33.768213 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" podStartSLOduration=1.76819799 podStartE2EDuration="1.76819799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:33.766312088 +0000 UTC m=+720.581506523" watchObservedRunningTime="2026-10-30 13:03:33.76819799 +0000 UTC m=+720.583392412" Oct 30 13:03:34.132943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.132909 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:34.190917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.190888 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk"] Oct 30 13:03:34.195090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.195067 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:03:34.228346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.211679 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d833bd-dockercfg-t5mnj\"" Oct 30 13:03:34.228346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.212410 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d833bd\"" Oct 30 13:03:34.228346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.227739 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk"] Oct 30 13:03:34.249090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.249056 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561552f7-0505-4fda-9fd1-eb1fa882a262-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk\" (UID: \"561552f7-0505-4fda-9fd1-eb1fa882a262\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:03:34.252910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.249377 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df5c2\" (UniqueName: \"kubernetes.io/projected/561552f7-0505-4fda-9fd1-eb1fa882a262-kube-api-access-df5c2\") pod \"payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk\" (UID: \"561552f7-0505-4fda-9fd1-eb1fa882a262\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:03:34.254228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.254101 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr"] Oct 30 13:03:34.258633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.258613 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:03:34.277064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.277032 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr"] Oct 30 13:03:34.283574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.283532 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:34.352042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.352011 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561552f7-0505-4fda-9fd1-eb1fa882a262-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk\" (UID: \"561552f7-0505-4fda-9fd1-eb1fa882a262\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:03:34.352240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.352096 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df5c2\" (UniqueName: \"kubernetes.io/projected/561552f7-0505-4fda-9fd1-eb1fa882a262-kube-api-access-df5c2\") pod \"payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk\" (UID: \"561552f7-0505-4fda-9fd1-eb1fa882a262\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:03:34.352240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.352169 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwsh\" (UniqueName: \"kubernetes.io/projected/14634916-26ab-4c10-bfba-03002f7a4eab-kube-api-access-wzwsh\") pod \"payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr\" (UID: \"14634916-26ab-4c10-bfba-03002f7a4eab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:03:34.352383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.352253 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14634916-26ab-4c10-bfba-03002f7a4eab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr\" (UID: \"14634916-26ab-4c10-bfba-03002f7a4eab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:03:34.353074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.353041 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561552f7-0505-4fda-9fd1-eb1fa882a262-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk\" (UID: \"561552f7-0505-4fda-9fd1-eb1fa882a262\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:03:34.377510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.377478 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df5c2\" (UniqueName: \"kubernetes.io/projected/561552f7-0505-4fda-9fd1-eb1fa882a262-kube-api-access-df5c2\") pod \"payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk\" (UID: \"561552f7-0505-4fda-9fd1-eb1fa882a262\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:03:34.453162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.453060 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwsh\" (UniqueName: \"kubernetes.io/projected/14634916-26ab-4c10-bfba-03002f7a4eab-kube-api-access-wzwsh\") pod \"payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr\" (UID: \"14634916-26ab-4c10-bfba-03002f7a4eab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:03:34.453162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.453131 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14634916-26ab-4c10-bfba-03002f7a4eab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr\" (UID: \"14634916-26ab-4c10-bfba-03002f7a4eab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:03:34.453883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.453863 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14634916-26ab-4c10-bfba-03002f7a4eab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr\" (UID: \"14634916-26ab-4c10-bfba-03002f7a4eab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:03:34.476809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.476783 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwsh\" (UniqueName: \"kubernetes.io/projected/14634916-26ab-4c10-bfba-03002f7a4eab-kube-api-access-wzwsh\") pod \"payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr\" (UID: \"14634916-26ab-4c10-bfba-03002f7a4eab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:03:34.540952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.540916 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:03:34.571268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.571236 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:03:34.858766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.858736 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk"] Oct 30 13:03:34.859307 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:03:34.859282 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561552f7_0505_4fda_9fd1_eb1fa882a262.slice/crio-07a66aa4ada2ddb89bdcb79a628bd9327b3f4b09912c03d89ac10e9e534abef2 WatchSource:0}: Error finding container 07a66aa4ada2ddb89bdcb79a628bd9327b3f4b09912c03d89ac10e9e534abef2: Status 404 returned error can't find the container with id 07a66aa4ada2ddb89bdcb79a628bd9327b3f4b09912c03d89ac10e9e534abef2 Oct 30 13:03:34.917550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.906067 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr"] Oct 30 13:03:34.989938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:34.989915 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:35.146289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.146152 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:35.636730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.636673 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:35.700856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.696971 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" event={"ID":"14634916-26ab-4c10-bfba-03002f7a4eab","Type":"ContainerStarted","Data":"9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c"} Oct 30 13:03:35.700856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.697077 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" event={"ID":"14634916-26ab-4c10-bfba-03002f7a4eab","Type":"ContainerStarted","Data":"fe46cb852ba694562d56322c904530ec00a1fdbc0e878adf795fcd814ad9d153"} Oct 30 13:03:35.700856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.698172 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:03:35.702740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.702713 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" event={"ID":"561552f7-0505-4fda-9fd1-eb1fa882a262","Type":"ContainerStarted","Data":"526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4"} Oct 30 13:03:35.702858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.702760 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" event={"ID":"561552f7-0505-4fda-9fd1-eb1fa882a262","Type":"ContainerStarted","Data":"07a66aa4ada2ddb89bdcb79a628bd9327b3f4b09912c03d89ac10e9e534abef2"} Oct 30 13:03:35.702902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.702849 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:03:35.755678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.755620 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" podStartSLOduration=1.755599305 podStartE2EDuration="1.755599305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:35.753542621 +0000 UTC m=+722.568737044" watchObservedRunningTime="2026-10-30 13:03:35.755599305 +0000 UTC m=+722.570793728" Oct 30 13:03:35.800128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.800099 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:35.800294 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:35.800134 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" podStartSLOduration=1.800121493 podStartE2EDuration="1.800121493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:35.799107523 +0000 UTC m=+722.614301947" watchObservedRunningTime="2026-10-30 13:03:35.800121493 +0000 UTC m=+722.615315915" Oct 30 13:03:37.183682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.183654 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:37.496993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.496893 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj"] Oct 30 13:03:37.501005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.500979 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:03:37.532350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.528651 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e34fd5\"" Oct 30 13:03:37.536731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.536700 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-e34fd5-dockercfg-jt5zf\"" Oct 30 13:03:37.544432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.544394 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj"] Oct 30 13:03:37.547935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.547905 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v"] Oct 30 13:03:37.548247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.548220 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:37.552671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.552649 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:03:37.567810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.567456 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v"] Oct 30 13:03:37.586929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.586895 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e5f412-4939-44cd-937d-2bf38707178e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v\" (UID: \"91e5f412-4939-44cd-937d-2bf38707178e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:03:37.587074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.586978 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55f9f0d5-52a1-434f-acb9-fdbfe8671335-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj\" (UID: \"55f9f0d5-52a1-434f-acb9-fdbfe8671335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:03:37.587132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.587074 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xng7h\" (UniqueName: \"kubernetes.io/projected/91e5f412-4939-44cd-937d-2bf38707178e-kube-api-access-xng7h\") pod \"payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v\" (UID: \"91e5f412-4939-44cd-937d-2bf38707178e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:03:37.587185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.587133 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86st8\" (UniqueName: \"kubernetes.io/projected/55f9f0d5-52a1-434f-acb9-fdbfe8671335-kube-api-access-86st8\") pod \"payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj\" (UID: \"55f9f0d5-52a1-434f-acb9-fdbfe8671335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:03:37.687961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.687924 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55f9f0d5-52a1-434f-acb9-fdbfe8671335-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj\" (UID: \"55f9f0d5-52a1-434f-acb9-fdbfe8671335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:03:37.688218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.688028 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng7h\" (UniqueName: \"kubernetes.io/projected/91e5f412-4939-44cd-937d-2bf38707178e-kube-api-access-xng7h\") pod \"payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v\" (UID: \"91e5f412-4939-44cd-937d-2bf38707178e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:03:37.688218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.688080 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86st8\" (UniqueName: \"kubernetes.io/projected/55f9f0d5-52a1-434f-acb9-fdbfe8671335-kube-api-access-86st8\") pod \"payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj\" (UID: \"55f9f0d5-52a1-434f-acb9-fdbfe8671335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:03:37.688218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.688131 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e5f412-4939-44cd-937d-2bf38707178e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v\" (UID: \"91e5f412-4939-44cd-937d-2bf38707178e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:03:37.689017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.688992 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e5f412-4939-44cd-937d-2bf38707178e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v\" (UID: \"91e5f412-4939-44cd-937d-2bf38707178e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:03:37.689181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.689163 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55f9f0d5-52a1-434f-acb9-fdbfe8671335-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj\" (UID: \"55f9f0d5-52a1-434f-acb9-fdbfe8671335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:03:37.735811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.735777 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86st8\" (UniqueName: \"kubernetes.io/projected/55f9f0d5-52a1-434f-acb9-fdbfe8671335-kube-api-access-86st8\") pod \"payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj\" (UID: \"55f9f0d5-52a1-434f-acb9-fdbfe8671335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:03:37.736430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.736403 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xng7h\" (UniqueName: \"kubernetes.io/projected/91e5f412-4939-44cd-937d-2bf38707178e-kube-api-access-xng7h\") pod \"payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v\" (UID: \"91e5f412-4939-44cd-937d-2bf38707178e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:03:37.827550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.827513 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:03:37.873679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:37.873650 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:03:38.178244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:38.178218 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj"] Oct 30 13:03:38.179937 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:03:38.179903 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f9f0d5_52a1_434f_acb9_fdbfe8671335.slice/crio-a1c56d5df53f72350893f377824e306b4d5b1f228e4d5588ce6793a0ce7c4035 WatchSource:0}: Error finding container a1c56d5df53f72350893f377824e306b4d5b1f228e4d5588ce6793a0ce7c4035: Status 404 returned error can't find the container with id a1c56d5df53f72350893f377824e306b4d5b1f228e4d5588ce6793a0ce7c4035 Oct 30 13:03:38.248339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:38.247978 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v"] Oct 30 13:03:38.248818 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:03:38.248338 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e5f412_4939_44cd_937d_2bf38707178e.slice/crio-d7d89d30c1d293b74742b11719a82badd27b4aeb039798f0d123967558304bcd WatchSource:0}: Error finding container d7d89d30c1d293b74742b11719a82badd27b4aeb039798f0d123967558304bcd: Status 404 returned error can't find the container with id d7d89d30c1d293b74742b11719a82badd27b4aeb039798f0d123967558304bcd Oct 30 13:03:38.751354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:38.749134 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" event={"ID":"91e5f412-4939-44cd-937d-2bf38707178e","Type":"ContainerStarted","Data":"21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b"} Oct 30 13:03:38.751354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:38.749384 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:03:38.751354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:38.749415 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" event={"ID":"91e5f412-4939-44cd-937d-2bf38707178e","Type":"ContainerStarted","Data":"d7d89d30c1d293b74742b11719a82badd27b4aeb039798f0d123967558304bcd"} Oct 30 13:03:38.751354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:38.750999 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" event={"ID":"55f9f0d5-52a1-434f-acb9-fdbfe8671335","Type":"ContainerStarted","Data":"2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4"} Oct 30 13:03:38.751354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:38.751044 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" event={"ID":"55f9f0d5-52a1-434f-acb9-fdbfe8671335","Type":"ContainerStarted","Data":"a1c56d5df53f72350893f377824e306b4d5b1f228e4d5588ce6793a0ce7c4035"} Oct 30 13:03:38.751354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:38.751153 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:03:38.791795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:38.791747 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" podStartSLOduration=1.791734624 podStartE2EDuration="1.791734624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:38.790109168 +0000 UTC m=+725.605303592" watchObservedRunningTime="2026-10-30 13:03:38.791734624 +0000 UTC m=+725.606929047" Oct 30 13:03:38.835013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:38.834965 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" podStartSLOduration=1.834950943 podStartE2EDuration="1.834950943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:38.833844754 +0000 UTC m=+725.649039178" watchObservedRunningTime="2026-10-30 13:03:38.834950943 +0000 UTC m=+725.650145365" Oct 30 13:03:39.216412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:39.216377 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:39.634887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:39.634858 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:03:39.639445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:39.639419 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:03:39.639586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:39.639465 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:41.086714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:41.086683 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:41.258834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:41.258774 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:41.444265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:41.444145 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:42.778625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:42.778583 5049 generic.go:356] "Generic (PLEG): container finished" podID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerID="1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442" exitCode=0 Oct 30 13:03:42.779071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:42.778654 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" event={"ID":"c7ad6423-a145-4a83-be4f-12fad5cb0a51","Type":"ContainerDied","Data":"1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442"} Oct 30 13:03:42.779071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:42.779054 5049 scope.go:122] "RemoveContainer" containerID="1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442" Oct 30 13:03:42.780454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:42.780427 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:42.780554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:42.780490 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:42.780554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:42.780518 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:43.364383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:43.344278 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:43.790263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:43.790169 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" event={"ID":"c7ad6423-a145-4a83-be4f-12fad5cb0a51","Type":"ContainerStarted","Data":"07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1"} Oct 30 13:03:44.103408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:44.103368 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:44.103620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:44.103488 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:03:44.103620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:44.103531 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 13:03:44.690234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:44.690196 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:03:44.690424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:44.690269 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:03:45.103860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.103823 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:03:45.104278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.103876 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 13:03:45.181347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.180725 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr"] Oct 30 13:03:45.185158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.185133 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr"] Oct 30 13:03:45.185471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.185425 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.219348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.206133 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zw9vx\"" Oct 30 13:03:45.219596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.206782 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:03:45.240344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.222400 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 13:03:45.240344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.222453 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-d7e62b-serving-cert\"" Oct 30 13:03:45.240344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.222508 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:03:45.502653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.498576 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.502653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.498671 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-metrics-certs\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.502653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.498761 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf87g\" (UniqueName: \"kubernetes.io/projected/58abbd14-8811-435e-b0f2-0826e0ba32eb-kube-api-access-kf87g\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.502653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.498951 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58abbd14-8811-435e-b0f2-0826e0ba32eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.502653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.501958 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:45.627364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.602888 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-metrics-certs\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.627364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.603009 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf87g\" (UniqueName: \"kubernetes.io/projected/58abbd14-8811-435e-b0f2-0826e0ba32eb-kube-api-access-kf87g\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.627364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.603124 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58abbd14-8811-435e-b0f2-0826e0ba32eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.627364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.603266 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.627364 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:03:45.603494 5049 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-serving-cert: secret "maas-api-e2e-worker-w4-d7e62b-serving-cert" not found Oct 30 13:03:45.627364 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:03:45.603574 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-maas-api-tls podName:58abbd14-8811-435e-b0f2-0826e0ba32eb nodeName:}" failed. No retries permitted until 2026-10-30 13:03:46.103544209 +0000 UTC m=+732.918738627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-maas-api-tls") pod "maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" (UID: "58abbd14-8811-435e-b0f2-0826e0ba32eb") : secret "maas-api-e2e-worker-w4-d7e62b-serving-cert" not found Oct 30 13:03:45.627364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.604585 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58abbd14-8811-435e-b0f2-0826e0ba32eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.627364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.619018 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-metrics-certs\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.666628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.658098 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf87g\" (UniqueName: \"kubernetes.io/projected/58abbd14-8811-435e-b0f2-0826e0ba32eb-kube-api-access-kf87g\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:45.685347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:45.683167 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:46.103966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:46.103930 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:03:46.104459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:46.103979 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 13:03:46.117044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:46.117004 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:46.142610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:46.142577 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:46.162170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:46.162121 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:46.163839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:46.163807 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:46.402419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:46.387333 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:46.791406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:46.752485 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:03:47.000614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:47.000577 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr"] Oct 30 13:03:47.029346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:47.005552 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:03:47.114489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:47.114448 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:03:47.114985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:47.114513 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 13:03:47.791605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:47.791576 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:03:48.095696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:48.095655 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:48.099281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:48.099241 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" event={"ID":"58abbd14-8811-435e-b0f2-0826e0ba32eb","Type":"ContainerStarted","Data":"ff9d28be0077dbd30e86fcea5f51b77c40db0155da8ea45d449b2f24925f463a"} Oct 30 13:03:48.104198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:48.104170 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:03:48.104360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:48.104214 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 13:03:48.269400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:48.266812 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:48.564033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:48.563958 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:49.104202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:49.103775 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:03:49.104202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:49.103837 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 13:03:49.761207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:49.761175 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:03:49.761987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:49.761782 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:03:50.108223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:50.108189 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:50.108991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:50.108938 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:50.109936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:50.109914 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:03:50.508610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:50.508525 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:52.171973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:52.171927 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" event={"ID":"58abbd14-8811-435e-b0f2-0826e0ba32eb","Type":"ContainerStarted","Data":"0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291"} Oct 30 13:03:52.172415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:52.172005 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:52.216564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:52.216494 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" podStartSLOduration=2.636772451 podStartE2EDuration="7.216474956s" totalImagesPullingTime="4.579702505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:52.215185925 +0000 UTC m=+739.030380349" watchObservedRunningTime="2026-10-30 13:03:52.216474956 +0000 UTC m=+739.031669381" Oct 30 13:03:52.397893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:52.397855 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:53.181224 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.181191 5049 generic.go:356] "Generic (PLEG): container finished" podID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerID="5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041" exitCode=0 Oct 30 13:03:53.181730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.181262 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" event={"ID":"06e20740-a55b-4ca5-8f06-f01b8b9cd57d","Type":"ContainerDied","Data":"5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041"} Oct 30 13:03:53.181730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.181695 5049 scope.go:122] "RemoveContainer" containerID="5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041" Oct 30 13:03:53.183179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.183147 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:53.183295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.183222 5049 generic.go:356] "Generic (PLEG): container finished" podID="87855fb7-df0f-468b-9b00-55f680e3a242" containerID="17a2c00c77578ef6aea3e7dcb338b55cfabf690e3bb4ecea302b29b97a116958" exitCode=0 Oct 30 13:03:53.183295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.183232 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:53.183295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.183278 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:53.183490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.183292 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" event={"ID":"87855fb7-df0f-468b-9b00-55f680e3a242","Type":"ContainerDied","Data":"17a2c00c77578ef6aea3e7dcb338b55cfabf690e3bb4ecea302b29b97a116958"} Oct 30 13:03:53.183900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.183825 5049 scope.go:122] "RemoveContainer" containerID="17a2c00c77578ef6aea3e7dcb338b55cfabf690e3bb4ecea302b29b97a116958" Oct 30 13:03:53.185024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.184998 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:53.185111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.185056 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:53.185111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:53.185087 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:54.197768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:54.197735 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" event={"ID":"87855fb7-df0f-468b-9b00-55f680e3a242","Type":"ContainerStarted","Data":"0aa25785191760e8c0dc85f46d8d8f26d0eb5d69731190d44934585eecb1c896"} Oct 30 13:03:54.200513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:54.200485 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" event={"ID":"06e20740-a55b-4ca5-8f06-f01b8b9cd57d","Type":"ContainerStarted","Data":"c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6"} Oct 30 13:03:54.438348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:54.433807 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:54.603889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:54.603854 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:54.793889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:54.793858 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:54.795175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:54.795152 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:03:54.795245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:54.795196 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 13:03:55.126346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:55.125664 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:03:55.126346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:55.125801 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 13:03:55.126346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:55.125842 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 13:03:55.615345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:55.611427 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:03:55.865344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:55.812805 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:03:55.865344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:55.812862 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 13:03:56.122778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:56.122737 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 13:03:56.123061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:56.123031 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 13:03:56.205759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:56.205723 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:56.751852 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:03:56.751815 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:03:56.793805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:56.793767 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:03:56.794084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:56.794054 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 13:03:56.960465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:56.959889 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:57.128756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:57.128720 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 13:03:57.129059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:57.129027 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 13:03:57.413394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:57.395852 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:57.755416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:57.753896 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:57.794779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:57.794737 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:03:57.795111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:57.795078 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 13:03:58.094347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:58.094188 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:58.124729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:58.124691 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 13:03:58.125019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:58.124989 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 13:03:58.194214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:58.194183 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:03:58.817347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:58.809534 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:03:58.817347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:58.809594 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 13:03:59.128409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:59.123727 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 13:03:59.128409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:59.123789 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 13:03:59.876711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:03:59.876670 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:00.104404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:00.104371 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:04:00.106117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:00.106094 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:04:00.154885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:00.154812 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:04:00.157674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:00.157646 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:04:00.897249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:00.897216 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:01.831437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:01.831402 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:01.992069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:01.992040 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:02.754602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:02.754570 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:02.922537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:02.922507 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:03.815840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:03.815806 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:04.520700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:04.520659 5049 generic.go:356] "Generic (PLEG): container finished" podID="9df030fd-edbd-4631-b7e5-292564e90d08" containerID="73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d" exitCode=0 Oct 30 13:04:04.520917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:04.520719 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" event={"ID":"9df030fd-edbd-4631-b7e5-292564e90d08","Type":"ContainerDied","Data":"73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d"} Oct 30 13:04:04.521176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:04.521158 5049 scope.go:122] "RemoveContainer" containerID="73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d" Oct 30 13:04:04.522534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:04.522506 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:04.522724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:04.522589 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:04.522724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:04.522641 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:05.537383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:05.537308 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" event={"ID":"9df030fd-edbd-4631-b7e5-292564e90d08","Type":"ContainerStarted","Data":"818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26"} Oct 30 13:04:05.741385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:05.737809 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:06.404835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:06.404799 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:04:06.428345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:06.417480 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:04:06.428345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:06.417566 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 13:04:06.552628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:06.552572 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:07.153860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:07.153822 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:07.386172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:07.386128 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:07.404460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:07.404374 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:04:07.404460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:07.404425 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 13:04:07.539259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:07.539219 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:07.753755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:07.753651 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:07.933172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:07.933137 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:08.404816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:08.404780 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:04:08.405133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:08.405092 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 13:04:09.240386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.218895 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r"] Oct 30 13:04:09.252633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.250662 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.256657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.256629 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r"] Oct 30 13:04:09.267611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.267589 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f07dfe\"/\"default-dockercfg-zhhrb\"" Oct 30 13:04:09.267611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.267593 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f07dfe\"/\"simulator-w0-f07dfe-kserve-self-signed-certs\"" Oct 30 13:04:09.272381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.272310 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f07dfe\"/\"kube-root-ca.crt\"" Oct 30 13:04:09.272641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.272349 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f07dfe\"/\"openshift-service-ca.crt\"" Oct 30 13:04:09.302513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.302474 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hxtx\" (UniqueName: \"kubernetes.io/projected/1d97d466-e892-4377-8c31-47dd077dbbc6-kube-api-access-9hxtx\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.302688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.302639 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-tmp-dir\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.302688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.302683 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-home\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.302789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.302722 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-model-cache\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.302857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.302830 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-dshm\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.302908 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.302892 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d97d466-e892-4377-8c31-47dd077dbbc6-tls-certs\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.332976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.332909 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:09.404172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.404135 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d97d466-e892-4377-8c31-47dd077dbbc6-tls-certs\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.404413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.404215 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hxtx\" (UniqueName: \"kubernetes.io/projected/1d97d466-e892-4377-8c31-47dd077dbbc6-kube-api-access-9hxtx\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.404553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.404522 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-tmp-dir\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.404640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.404594 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-home\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.404640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.404621 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-model-cache\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.404640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.404625 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:04:09.404794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.404649 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-dshm\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.404794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.404677 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 13:04:09.405039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.405017 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-model-cache\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.405130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.405043 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-home\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.405130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.405121 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-tmp-dir\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.432344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.432009 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d97d466-e892-4377-8c31-47dd077dbbc6-tls-certs\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.432344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.432291 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-dshm\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.436578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.436549 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hxtx\" (UniqueName: \"kubernetes.io/projected/1d97d466-e892-4377-8c31-47dd077dbbc6-kube-api-access-9hxtx\") pod \"simulator-w0-f07dfe-kserve-848887bb7f-6r62r\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.569329 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.569285 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:09.603628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.603596 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:04:09.898642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:09.898482 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r"] Oct 30 13:04:09.901077 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:04:09.901047 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d97d466_e892_4377_8c31_47dd077dbbc6.slice/crio-1df40a181fcc193a53dd77ad5342b821a82acfc787f04664948e0f61c3783443 WatchSource:0}: Error finding container 1df40a181fcc193a53dd77ad5342b821a82acfc787f04664948e0f61c3783443: Status 404 returned error can't find the container with id 1df40a181fcc193a53dd77ad5342b821a82acfc787f04664948e0f61c3783443 Oct 30 13:04:10.044480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:10.044447 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:10.404449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:10.404410 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:04:10.404849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:10.404467 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 13:04:10.579550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:10.579508 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" event={"ID":"1d97d466-e892-4377-8c31-47dd077dbbc6","Type":"ContainerStarted","Data":"6ed7b73faaa0fe3b49a4c6561ce8983b52bc42832148677ca8a1a1699b17b3c2"} Oct 30 13:04:10.579550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:10.579567 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" event={"ID":"1d97d466-e892-4377-8c31-47dd077dbbc6","Type":"ContainerStarted","Data":"1df40a181fcc193a53dd77ad5342b821a82acfc787f04664948e0f61c3783443"} Oct 30 13:04:10.627192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:10.627130 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podStartSLOduration=1.62711137 podStartE2EDuration="1.62711137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:10.624375812 +0000 UTC m=+757.439570238" watchObservedRunningTime="2026-10-30 13:04:10.62711137 +0000 UTC m=+757.442305794" Oct 30 13:04:11.409270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:11.409241 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:04:11.410337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:11.410300 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:04:11.569961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:11.569931 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:12.011833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:12.011794 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:12.749335 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:12.749280 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:13.384769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:13.384733 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:13.563710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:13.563667 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:13.791311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:13.791226 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:14.057029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:14.056992 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:16.205078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:16.205046 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:17.652061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.652025 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx"] Oct 30 13:04:17.656050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.656031 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.682295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.682272 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"kube-root-ca.crt\"" Oct 30 13:04:17.682448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.682331 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"openshift-service-ca.crt\"" Oct 30 13:04:17.682448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.682345 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"default-dockercfg-z4nxf\"" Oct 30 13:04:17.682448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.682443 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"simulator-w2-cb58b8-kserve-self-signed-certs\"" Oct 30 13:04:17.684913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.684892 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx"] Oct 30 13:04:17.727188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.727145 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqjh\" (UniqueName: \"kubernetes.io/projected/f7747aab-1a05-402a-b859-9d4afa588683-kube-api-access-bfqjh\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.727388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.727215 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7747aab-1a05-402a-b859-9d4afa588683-tls-certs\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.727388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.727237 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-dshm\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.727388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.727252 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-home\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.727498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.727404 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-tmp-dir\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.727498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.727458 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-model-cache\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.774270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.774242 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:17.828433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.828400 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-tmp-dir\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.828605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.828477 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-model-cache\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.828605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.828576 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqjh\" (UniqueName: \"kubernetes.io/projected/f7747aab-1a05-402a-b859-9d4afa588683-kube-api-access-bfqjh\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.828734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.828629 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7747aab-1a05-402a-b859-9d4afa588683-tls-certs\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.828795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.828777 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-dshm\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.828849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.828807 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-home\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.828849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.828822 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-tmp-dir\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.828950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.828899 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-model-cache\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.829114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.829091 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-home\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.830977 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.830953 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-dshm\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.831181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.831166 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7747aab-1a05-402a-b859-9d4afa588683-tls-certs\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.857875 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.857851 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqjh\" (UniqueName: \"kubernetes.io/projected/f7747aab-1a05-402a-b859-9d4afa588683-kube-api-access-bfqjh\") pod \"simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:17.970496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:17.970393 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:18.226392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:18.226282 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:18.296814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:18.296789 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx"] Oct 30 13:04:18.299409 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:04:18.299381 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7747aab_1a05_402a_b859_9d4afa588683.slice/crio-b28ce877ebebf587a3872ea7636845e01efddc6dfce46c2d0cbce26201f33905 WatchSource:0}: Error finding container b28ce877ebebf587a3872ea7636845e01efddc6dfce46c2d0cbce26201f33905: Status 404 returned error can't find the container with id b28ce877ebebf587a3872ea7636845e01efddc6dfce46c2d0cbce26201f33905 Oct 30 13:04:18.640055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:18.640009 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" event={"ID":"f7747aab-1a05-402a-b859-9d4afa588683","Type":"ContainerStarted","Data":"1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625"} Oct 30 13:04:18.640247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:18.640066 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" event={"ID":"f7747aab-1a05-402a-b859-9d4afa588683","Type":"ContainerStarted","Data":"b28ce877ebebf587a3872ea7636845e01efddc6dfce46c2d0cbce26201f33905"} Oct 30 13:04:18.691955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:18.691902 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podStartSLOduration=1.6918761519999999 podStartE2EDuration="1.691876152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:18.688149123 +0000 UTC m=+765.503343546" watchObservedRunningTime="2026-10-30 13:04:18.691876152 +0000 UTC m=+765.507070572" Oct 30 13:04:18.970990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:18.970892 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:19.570400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:19.570360 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:19.583403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:19.583372 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:19.660863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:19.660832 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:04:20.102573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:20.102516 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:21.980349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:21.979886 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:23.998428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:23.998387 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:24.649840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:24.649812 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:24.799088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:24.799050 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:25.969824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:25.969783 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:26.677063 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:04:26.677028 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:04:27.781667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:27.781618 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:27.971461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:27.971424 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:27.984282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:27.984248 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:28.720827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:28.720792 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:04:29.894283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:29.894239 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:31.772658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:31.772625 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:32.281806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.281769 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m"] Oct 30 13:04:32.286842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.286815 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.311896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.311868 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"default-dockercfg-slxg9\"" Oct 30 13:04:32.340097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.324550 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"premium-w1-52a649-kserve-self-signed-certs\"" Oct 30 13:04:32.340097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.330391 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"kube-root-ca.crt\"" Oct 30 13:04:32.340097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.330446 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"openshift-service-ca.crt\"" Oct 30 13:04:32.340352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.340200 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m"] Oct 30 13:04:32.375044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.375011 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:32.385043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.385001 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-dshm\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.385199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.385093 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hld64\" (UniqueName: \"kubernetes.io/projected/ea6ab97e-bb42-458a-8a78-f8752042da2e-kube-api-access-hld64\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.385199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.385149 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-model-cache\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.385281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.385228 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-tmp-dir\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.385353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.385309 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-home\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.385399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.385372 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6ab97e-bb42-458a-8a78-f8752042da2e-tls-certs\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.486569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.486528 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6ab97e-bb42-458a-8a78-f8752042da2e-tls-certs\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.486786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.486630 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-dshm\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.486786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.486665 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hld64\" (UniqueName: \"kubernetes.io/projected/ea6ab97e-bb42-458a-8a78-f8752042da2e-kube-api-access-hld64\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.486786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.486711 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-model-cache\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.486786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.486737 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-tmp-dir\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.486987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.486787 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-home\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.487185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.487161 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-home\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.487491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.487460 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-model-cache\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.487773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.487747 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-tmp-dir\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.489167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.489139 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6ab97e-bb42-458a-8a78-f8752042da2e-tls-certs\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.489613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.489595 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-dshm\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.509701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.509667 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hld64\" (UniqueName: \"kubernetes.io/projected/ea6ab97e-bb42-458a-8a78-f8752042da2e-kube-api-access-hld64\") pod \"premium-w1-52a649-kserve-58ff6dc548-bxt4m\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.601501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.601467 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:32.902671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:32.902640 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m"] Oct 30 13:04:32.904796 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:04:32.904765 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea6ab97e_bb42_458a_8a78_f8752042da2e.slice/crio-af08d9cb677773c346afe9d851d3bb26e601a8da533ac96e99899bc5d17401f9 WatchSource:0}: Error finding container af08d9cb677773c346afe9d851d3bb26e601a8da533ac96e99899bc5d17401f9: Status 404 returned error can't find the container with id af08d9cb677773c346afe9d851d3bb26e601a8da533ac96e99899bc5d17401f9 Oct 30 13:04:33.532431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:33.532386 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:33.734091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:33.734053 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" event={"ID":"ea6ab97e-bb42-458a-8a78-f8752042da2e","Type":"ContainerStarted","Data":"6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e"} Oct 30 13:04:33.734277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:33.734102 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" event={"ID":"ea6ab97e-bb42-458a-8a78-f8752042da2e","Type":"ContainerStarted","Data":"af08d9cb677773c346afe9d851d3bb26e601a8da533ac96e99899bc5d17401f9"} Oct 30 13:04:33.775496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:33.775447 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podStartSLOduration=1.775434122 podStartE2EDuration="1.775434122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:33.773207732 +0000 UTC m=+780.588402155" watchObservedRunningTime="2026-10-30 13:04:33.775434122 +0000 UTC m=+780.590628541" Oct 30 13:04:34.090086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:34.090045 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:34.601585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:34.601555 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:35.604592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:35.604561 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:37.207040 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.207001 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2"] Oct 30 13:04:37.211817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.211791 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.228403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.228377 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ee7d65-openshift-default-dockercfg-5cfmj\"" Oct 30 13:04:37.239507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.239458 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.252353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.249524 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.252792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.252577 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.252987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.252965 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.253147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.253119 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.253268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.253175 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.253268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.253255 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.253427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.253311 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.253427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.253395 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znbg\" (UniqueName: \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-kube-api-access-4znbg\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.257851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.257823 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2"] Oct 30 13:04:37.354827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.354784 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.355023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.354851 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.355023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.354902 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znbg\" (UniqueName: \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-kube-api-access-4znbg\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.355023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.354939 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.355023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.354988 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.355228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.355027 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.355228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.355067 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.355228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.355108 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.355228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.355139 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.355781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.355751 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.356526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.356476 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.356680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.356656 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.356816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.356795 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.357034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.357009 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.357718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.357698 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.358302 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.358274 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.378265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.378230 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.378485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.378462 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znbg\" (UniqueName: \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-kube-api-access-4znbg\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.529490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.529406 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:37.579639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.579600 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:37.915887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.915491 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2"] Oct 30 13:04:37.923352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.923283 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:37.923556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.923533 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:37.923703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:37.923684 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:38.771829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:38.770053 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" event={"ID":"1f0bd8bb-ae7a-40ad-950c-058013605dd6","Type":"ContainerStarted","Data":"91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7"} Oct 30 13:04:38.771829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:38.770116 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" event={"ID":"1f0bd8bb-ae7a-40ad-950c-058013605dd6","Type":"ContainerStarted","Data":"1fc63f16b79f3aebeb156d53edf55454b1051078fa009195c8938db840e347fc"} Oct 30 13:04:38.849249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:38.849190 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" podStartSLOduration=1.849167773 podStartE2EDuration="1.849167773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:38.829793211 +0000 UTC m=+785.644987647" watchObservedRunningTime="2026-10-30 13:04:38.849167773 +0000 UTC m=+785.664362197" Oct 30 13:04:39.542380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:39.539461 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:39.542380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:39.539612 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:04:39.542380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:39.539658 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:04:39.683813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:39.683779 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:40.533740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:40.533696 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:04:40.534352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:40.534292 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:04:41.347400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:41.347365 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:41.530242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:41.530208 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:04:41.530528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:41.530502 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:04:41.646739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:41.646632 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:42.531393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:42.531310 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:04:42.531577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:42.531421 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:04:42.602627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:42.602592 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:42.643256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:42.643223 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:42.826606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:42.826575 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:04:43.498085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:43.498048 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:43.532754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:43.532715 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:04:43.533036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:43.533007 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:04:44.692150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:44.692118 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:44.692603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:44.692577 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:44.694607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:44.694584 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:04:45.414800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:45.414747 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:47.267833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:47.267796 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:49.088274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:49.088243 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:51.015645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:51.015610 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:52.676719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.676678 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn"] Oct 30 13:04:52.681383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.681359 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.697114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.697088 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f07dfe\"/\"premium-w0-f07dfe-kserve-self-signed-certs\"" Oct 30 13:04:52.711603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.711572 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn"] Oct 30 13:04:52.818082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.818044 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:52.855981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.855946 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-dshm\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.856139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.855982 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-tmp-dir\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.856139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.856118 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e59bea39-c435-4272-b9e2-ee38897a3f55-tls-certs\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.856273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.856255 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-home\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.856378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.856357 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c96c\" (UniqueName: \"kubernetes.io/projected/e59bea39-c435-4272-b9e2-ee38897a3f55-kube-api-access-5c96c\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.856471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.856452 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-model-cache\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.957227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.957140 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-dshm\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.957227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.957178 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-tmp-dir\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.957227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.957226 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e59bea39-c435-4272-b9e2-ee38897a3f55-tls-certs\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.957490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.957287 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-home\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.957490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.957333 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c96c\" (UniqueName: \"kubernetes.io/projected/e59bea39-c435-4272-b9e2-ee38897a3f55-kube-api-access-5c96c\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.957490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.957389 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-model-cache\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.957741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.957718 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-tmp-dir\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.957836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.957790 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-model-cache\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.957896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.957854 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-home\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.959500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.959482 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-dshm\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.959779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.959761 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e59bea39-c435-4272-b9e2-ee38897a3f55-tls-certs\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.981724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.981695 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c96c\" (UniqueName: \"kubernetes.io/projected/e59bea39-c435-4272-b9e2-ee38897a3f55-kube-api-access-5c96c\") pod \"premium-w0-f07dfe-kserve-98dff7894-xnchn\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:52.995732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:52.995708 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:53.006017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:53.005960 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:53.305514 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:04:53.305484 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode59bea39_c435_4272_b9e2_ee38897a3f55.slice/crio-ffeebd0a344bb73acdb543630ff207fc22c320c3ae2942f0b6c69235e893747e WatchSource:0}: Error finding container ffeebd0a344bb73acdb543630ff207fc22c320c3ae2942f0b6c69235e893747e: Status 404 returned error can't find the container with id ffeebd0a344bb73acdb543630ff207fc22c320c3ae2942f0b6c69235e893747e Oct 30 13:04:53.306303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:53.306278 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn"] Oct 30 13:04:53.870928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:53.870891 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" event={"ID":"e59bea39-c435-4272-b9e2-ee38897a3f55","Type":"ContainerStarted","Data":"92fb3846104009974a50456818e00bd02b6699409c6a197178b7571a4ce667a7"} Oct 30 13:04:53.871315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:53.870937 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" event={"ID":"e59bea39-c435-4272-b9e2-ee38897a3f55","Type":"ContainerStarted","Data":"ffeebd0a344bb73acdb543630ff207fc22c320c3ae2942f0b6c69235e893747e"} Oct 30 13:04:53.910248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:53.910191 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podStartSLOduration=1.910177764 podStartE2EDuration="1.910177764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:53.908717542 +0000 UTC m=+800.723912041" watchObservedRunningTime="2026-10-30 13:04:53.910177764 +0000 UTC m=+800.725372201" Oct 30 13:04:53.996841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:53.996811 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:04:54.629710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:54.629676 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:56.527598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:56.527554 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:56.676720 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:04:56.676686 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:04:58.380180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:58.380115 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:58.661169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:04:58.661080 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:00.159418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:00.159287 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:02.025346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:02.016152 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:02.197181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:02.197143 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:02.996421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:02.996383 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:05:03.010940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:03.010911 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:05:03.951973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:03.951941 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:03.995553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:03.995527 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:05:05.423184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.423149 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf"] Oct 30 13:05:05.427663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.427636 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.433664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.433635 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381f988e-0231-413c-950d-2302ce2058c2-tls-certs\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.433811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.433672 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-model-cache\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.433811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.433702 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-home\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.433811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.433799 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-dshm\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.433968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.433825 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-tmp-dir\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.434042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.434021 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbf9\" (UniqueName: \"kubernetes.io/projected/381f988e-0231-413c-950d-2302ce2058c2-kube-api-access-wtbf9\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.442692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.442671 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"kube-root-ca.crt\"" Oct 30 13:05:05.453228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.453201 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"premium-w4-d7e62b-kserve-self-signed-certs\"" Oct 30 13:05:05.459259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.459229 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf"] Oct 30 13:05:05.535377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.535341 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbf9\" (UniqueName: \"kubernetes.io/projected/381f988e-0231-413c-950d-2302ce2058c2-kube-api-access-wtbf9\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.535589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.535429 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381f988e-0231-413c-950d-2302ce2058c2-tls-certs\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.535589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.535451 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-model-cache\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.535589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.535469 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-home\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.535589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.535517 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-dshm\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.535589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.535546 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-tmp-dir\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.535988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.535952 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-model-cache\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.536076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.535992 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-tmp-dir\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.536220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.536200 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-home\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.538239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.538218 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-dshm\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.538631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.538613 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381f988e-0231-413c-950d-2302ce2058c2-tls-certs\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.543007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.542988 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"openshift-service-ca.crt\"" Oct 30 13:05:05.546271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.546253 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"default-dockercfg-x76nv\"" Oct 30 13:05:05.562445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.562415 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbf9\" (UniqueName: \"kubernetes.io/projected/381f988e-0231-413c-950d-2302ce2058c2-kube-api-access-wtbf9\") pod \"premium-w4-d7e62b-kserve-5cb796764d-2zjgf\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.571462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.571438 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:05.742093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.742007 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:05.819353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:05.819027 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:06.050634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:06.050604 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf"] Oct 30 13:05:06.053317 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:06.053288 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod381f988e_0231_413c_950d_2302ce2058c2.slice/crio-cb3e5f385836edf91debcff20e750afb215aa0c15be2b078a494bf94e239a61b WatchSource:0}: Error finding container cb3e5f385836edf91debcff20e750afb215aa0c15be2b078a494bf94e239a61b: Status 404 returned error can't find the container with id cb3e5f385836edf91debcff20e750afb215aa0c15be2b078a494bf94e239a61b Oct 30 13:05:07.006661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:07.006626 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" event={"ID":"381f988e-0231-413c-950d-2302ce2058c2","Type":"ContainerStarted","Data":"7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f"} Oct 30 13:05:07.006661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:07.006667 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" event={"ID":"381f988e-0231-413c-950d-2302ce2058c2","Type":"ContainerStarted","Data":"cb3e5f385836edf91debcff20e750afb215aa0c15be2b078a494bf94e239a61b"} Oct 30 13:05:07.052783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:07.052720 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podStartSLOduration=2.0527028 podStartE2EDuration="2.0527028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:07.050871651 +0000 UTC m=+813.866066074" watchObservedRunningTime="2026-10-30 13:05:07.0527028 +0000 UTC m=+813.867897224" Oct 30 13:05:07.593983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:07.593945 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:07.742470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:07.742441 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:09.375850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.375816 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw"] Oct 30 13:05:09.379900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.379880 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:09.392849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.392826 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:09.403225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.403195 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:05:09.406215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.406188 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw"] Oct 30 13:05:09.443938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.443423 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:09.482238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.482207 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699t6\" (UniqueName: \"kubernetes.io/projected/c2a04399-e68c-4978-981b-f2987aec0e16-kube-api-access-699t6\") pod \"maas-api-revoke-sub-3037e700-69drw\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:09.482452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.482243 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3037e700-69drw\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:09.482502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.482467 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3037e700-69drw\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:09.583283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.583252 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3037e700-69drw\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:09.583482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.583300 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699t6\" (UniqueName: \"kubernetes.io/projected/c2a04399-e68c-4978-981b-f2987aec0e16-kube-api-access-699t6\") pod \"maas-api-revoke-sub-3037e700-69drw\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:09.583482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.583354 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3037e700-69drw\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:09.584086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.584059 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3037e700-69drw\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:09.585607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.585583 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3037e700-69drw\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:09.604703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.604674 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-699t6\" (UniqueName: \"kubernetes.io/projected/c2a04399-e68c-4978-981b-f2987aec0e16-kube-api-access-699t6\") pod \"maas-api-revoke-sub-3037e700-69drw\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:09.693517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:09.693437 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:10.009849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:10.009818 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw"] Oct 30 13:05:10.011377 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:10.011344 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a04399_e68c_4978_981b_f2987aec0e16.slice/crio-fb0e31fd3d578f3fbdb59a219fda9106f31c5e554b7c1d77a56f7d9375b5676b WatchSource:0}: Error finding container fb0e31fd3d578f3fbdb59a219fda9106f31c5e554b7c1d77a56f7d9375b5676b: Status 404 returned error can't find the container with id fb0e31fd3d578f3fbdb59a219fda9106f31c5e554b7c1d77a56f7d9375b5676b Oct 30 13:05:10.031822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:10.031793 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" event={"ID":"c2a04399-e68c-4978-981b-f2987aec0e16","Type":"ContainerStarted","Data":"fb0e31fd3d578f3fbdb59a219fda9106f31c5e554b7c1d77a56f7d9375b5676b"} Oct 30 13:05:11.351939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:11.351879 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:13.056525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:13.056485 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" event={"ID":"c2a04399-e68c-4978-981b-f2987aec0e16","Type":"ContainerStarted","Data":"f213f1526dd1f8da8995e625ce3a3a75745d67338a26a37690e070f98b28eeae"} Oct 30 13:05:13.104807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:13.104752 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" podStartSLOduration=1.6389285930000002 podStartE2EDuration="4.104738895s" totalImagesPullingTime="2.465810302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:13.102210434 +0000 UTC m=+819.917404858" watchObservedRunningTime="2026-10-30 13:05:13.104738895 +0000 UTC m=+819.919933317" Oct 30 13:05:13.559344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:13.549236 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:14.066898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:14.066854 5049 generic.go:356] "Generic (PLEG): container finished" podID="c2a04399-e68c-4978-981b-f2987aec0e16" containerID="f213f1526dd1f8da8995e625ce3a3a75745d67338a26a37690e070f98b28eeae" exitCode=0 Oct 30 13:05:14.067364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:14.066994 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" event={"ID":"c2a04399-e68c-4978-981b-f2987aec0e16","Type":"ContainerDied","Data":"f213f1526dd1f8da8995e625ce3a3a75745d67338a26a37690e070f98b28eeae"} Oct 30 13:05:14.645682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:14.645645 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:15.233164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.233136 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:15.374396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.374306 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-openshift-ca\") pod \"c2a04399-e68c-4978-981b-f2987aec0e16\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " Oct 30 13:05:15.374530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.374425 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a04399-e68c-4978-981b-f2987aec0e16-kube-api-access-699t6\" (UniqueName: \"kubernetes.io/projected/c2a04399-e68c-4978-981b-f2987aec0e16-kube-api-access-699t6\") pod \"c2a04399-e68c-4978-981b-f2987aec0e16\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " Oct 30 13:05:15.374530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.374493 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-xks-ca\") pod \"c2a04399-e68c-4978-981b-f2987aec0e16\" (UID: \"c2a04399-e68c-4978-981b-f2987aec0e16\") " Oct 30 13:05:15.374726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.374697 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-openshift-ca" pod "c2a04399-e68c-4978-981b-f2987aec0e16" (UID: "c2a04399-e68c-4978-981b-f2987aec0e16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:15.374967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.374945 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:15.376692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.376663 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a04399-e68c-4978-981b-f2987aec0e16-kube-api-access-699t6" pod "c2a04399-e68c-4978-981b-f2987aec0e16" (UID: "c2a04399-e68c-4978-981b-f2987aec0e16"). InnerVolumeSpecName "kube-api-access-699t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:15.376795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.376689 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-xks-ca" pod "c2a04399-e68c-4978-981b-f2987aec0e16" (UID: "c2a04399-e68c-4978-981b-f2987aec0e16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:15.437529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.437490 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:15.476206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.476173 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-699t6\" (UniqueName: \"kubernetes.io/projected/c2a04399-e68c-4978-981b-f2987aec0e16-kube-api-access-699t6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:15.476206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.476208 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a04399-e68c-4978-981b-f2987aec0e16-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:15.742302 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.742220 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:15.755049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:15.755024 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:16.080393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:16.080368 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" Oct 30 13:05:16.080581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:16.080361 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw" event={"ID":"c2a04399-e68c-4978-981b-f2987aec0e16","Type":"ContainerDied","Data":"fb0e31fd3d578f3fbdb59a219fda9106f31c5e554b7c1d77a56f7d9375b5676b"} Oct 30 13:05:16.080581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:16.080493 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb0e31fd3d578f3fbdb59a219fda9106f31c5e554b7c1d77a56f7d9375b5676b" Oct 30 13:05:16.093365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:16.093342 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:05:17.481163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:17.481129 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:19.434931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:19.434897 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:21.182010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:21.181972 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:23.096357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:23.096304 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:24.678547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:24.678511 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:25.089221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:25.089185 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:26.683498 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:05:26.683461 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:05:26.999289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:26.999203 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:29.000584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.000542 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:29.862075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.862035 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch"] Oct 30 13:05:29.862910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.862882 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c2a04399-e68c-4978-981b-f2987aec0e16" containerName="revoke-keys" Oct 30 13:05:29.862910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.862907 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a04399-e68c-4978-981b-f2987aec0e16" containerName="revoke-keys" Oct 30 13:05:29.863078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.863066 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a04399-e68c-4978-981b-f2987aec0e16" containerName="revoke-keys" Oct 30 13:05:29.869045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.869026 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:29.873976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.873948 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctv7q\" (UniqueName: \"kubernetes.io/projected/aec47a3e-9af4-4396-bc97-85c8ae728b92-kube-api-access-ctv7q\") pod \"maas-api-revoke-sub-81f1e5fd-ht4ch\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:29.874077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.874003 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f1e5fd-ht4ch\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:29.874196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.874175 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f1e5fd-ht4ch\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:29.885788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.885764 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch"] Oct 30 13:05:29.893482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.893455 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:29.893669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.893650 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:05:29.975792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.975762 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctv7q\" (UniqueName: \"kubernetes.io/projected/aec47a3e-9af4-4396-bc97-85c8ae728b92-kube-api-access-ctv7q\") pod \"maas-api-revoke-sub-81f1e5fd-ht4ch\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:29.975990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.975801 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f1e5fd-ht4ch\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:29.975990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.975884 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f1e5fd-ht4ch\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:29.976698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.976670 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f1e5fd-ht4ch\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:29.978430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.978399 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f1e5fd-ht4ch\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:29.997332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:29.997300 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctv7q\" (UniqueName: \"kubernetes.io/projected/aec47a3e-9af4-4396-bc97-85c8ae728b92-kube-api-access-ctv7q\") pod \"maas-api-revoke-sub-81f1e5fd-ht4ch\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:30.183548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:30.183460 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:30.492194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:30.492167 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch"] Oct 30 13:05:30.494524 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:30.494497 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec47a3e_9af4_4396_bc97_85c8ae728b92.slice/crio-770cb710be00732d17f270cc635b460226f1f00f4653f1432c3e1ac1b1d2ebae WatchSource:0}: Error finding container 770cb710be00732d17f270cc635b460226f1f00f4653f1432c3e1ac1b1d2ebae: Status 404 returned error can't find the container with id 770cb710be00732d17f270cc635b460226f1f00f4653f1432c3e1ac1b1d2ebae Oct 30 13:05:30.791724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:30.791629 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:31.226824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:31.226787 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" event={"ID":"aec47a3e-9af4-4396-bc97-85c8ae728b92","Type":"ContainerStarted","Data":"cf7c03a72fb8758df70107ecd8a9d4aeb8b2829b4ea84bdfc38565a4035405bb"} Oct 30 13:05:31.227242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:31.226886 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" event={"ID":"aec47a3e-9af4-4396-bc97-85c8ae728b92","Type":"ContainerStarted","Data":"770cb710be00732d17f270cc635b460226f1f00f4653f1432c3e1ac1b1d2ebae"} Oct 30 13:05:31.266777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:31.266708 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" podStartSLOduration=2.266687154 podStartE2EDuration="2.266687154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:31.264597455 +0000 UTC m=+838.079791878" watchObservedRunningTime="2026-10-30 13:05:31.266687154 +0000 UTC m=+838.081881583" Oct 30 13:05:32.235509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:32.235476 5049 generic.go:356] "Generic (PLEG): container finished" podID="aec47a3e-9af4-4396-bc97-85c8ae728b92" containerID="cf7c03a72fb8758df70107ecd8a9d4aeb8b2829b4ea84bdfc38565a4035405bb" exitCode=0 Oct 30 13:05:32.235938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:32.235552 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" event={"ID":"aec47a3e-9af4-4396-bc97-85c8ae728b92","Type":"ContainerDied","Data":"cf7c03a72fb8758df70107ecd8a9d4aeb8b2829b4ea84bdfc38565a4035405bb"} Oct 30 13:05:32.766582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:32.766544 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:32.839214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:32.839172 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx"] Oct 30 13:05:32.843460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:32.843437 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:32.865421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:32.865393 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx"] Oct 30 13:05:32.907215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:32.907180 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ff3c51a-kndmx\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:32.907432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:32.907298 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2pbs\" (UniqueName: \"kubernetes.io/projected/1231f0e6-4e20-4f34-b004-b7a284cf26d8-kube-api-access-s2pbs\") pod \"maas-api-revoke-sub-9ff3c51a-kndmx\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:32.907432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:32.907387 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ff3c51a-kndmx\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:33.008919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.008887 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ff3c51a-kndmx\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:33.009113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.008968 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2pbs\" (UniqueName: \"kubernetes.io/projected/1231f0e6-4e20-4f34-b004-b7a284cf26d8-kube-api-access-s2pbs\") pod \"maas-api-revoke-sub-9ff3c51a-kndmx\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:33.009113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.009010 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ff3c51a-kndmx\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:33.009664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.009637 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ff3c51a-kndmx\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:33.011332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.011288 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ff3c51a-kndmx\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:33.030746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.030674 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2pbs\" (UniqueName: \"kubernetes.io/projected/1231f0e6-4e20-4f34-b004-b7a284cf26d8-kube-api-access-s2pbs\") pod \"maas-api-revoke-sub-9ff3c51a-kndmx\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:33.159945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.159913 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:33.407975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.407946 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:33.467825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.467706 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx"] Oct 30 13:05:33.470826 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:33.470795 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1231f0e6_4e20_4f34_b004_b7a284cf26d8.slice/crio-f219db3517abd3ba7f777b597d86586a902f75f7244d4eba083c7da27a3a1c83 WatchSource:0}: Error finding container f219db3517abd3ba7f777b597d86586a902f75f7244d4eba083c7da27a3a1c83: Status 404 returned error can't find the container with id f219db3517abd3ba7f777b597d86586a902f75f7244d4eba083c7da27a3a1c83 Oct 30 13:05:33.523149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.523118 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec47a3e-9af4-4396-bc97-85c8ae728b92-kube-api-access-ctv7q\" (UniqueName: \"kubernetes.io/projected/aec47a3e-9af4-4396-bc97-85c8ae728b92-kube-api-access-ctv7q\") pod \"aec47a3e-9af4-4396-bc97-85c8ae728b92\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " Oct 30 13:05:33.523292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.523176 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-xks-ca\") pod \"aec47a3e-9af4-4396-bc97-85c8ae728b92\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " Oct 30 13:05:33.523292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.523205 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-openshift-ca\") pod \"aec47a3e-9af4-4396-bc97-85c8ae728b92\" (UID: \"aec47a3e-9af4-4396-bc97-85c8ae728b92\") " Oct 30 13:05:33.523693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.523672 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-openshift-ca" pod "aec47a3e-9af4-4396-bc97-85c8ae728b92" (UID: "aec47a3e-9af4-4396-bc97-85c8ae728b92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:33.525304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.525278 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec47a3e-9af4-4396-bc97-85c8ae728b92-kube-api-access-ctv7q" pod "aec47a3e-9af4-4396-bc97-85c8ae728b92" (UID: "aec47a3e-9af4-4396-bc97-85c8ae728b92"). InnerVolumeSpecName "kube-api-access-ctv7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:33.525475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.525453 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-xks-ca" pod "aec47a3e-9af4-4396-bc97-85c8ae728b92" (UID: "aec47a3e-9af4-4396-bc97-85c8ae728b92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:33.624670 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.624593 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctv7q\" (UniqueName: \"kubernetes.io/projected/aec47a3e-9af4-4396-bc97-85c8ae728b92-kube-api-access-ctv7q\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:33.624670 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.624619 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:33.624670 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:33.624628 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec47a3e-9af4-4396-bc97-85c8ae728b92-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:34.254116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:34.254080 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" Oct 30 13:05:34.254381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:34.254111 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch" event={"ID":"aec47a3e-9af4-4396-bc97-85c8ae728b92","Type":"ContainerDied","Data":"770cb710be00732d17f270cc635b460226f1f00f4653f1432c3e1ac1b1d2ebae"} Oct 30 13:05:34.254381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:34.254170 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="770cb710be00732d17f270cc635b460226f1f00f4653f1432c3e1ac1b1d2ebae" Oct 30 13:05:34.255856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:34.255827 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" event={"ID":"1231f0e6-4e20-4f34-b004-b7a284cf26d8","Type":"ContainerStarted","Data":"d363d56ac9bd33377bd2f213f410dfb36bcaea11744554ad8a4ed78334547afc"} Oct 30 13:05:34.255991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:34.255876 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" event={"ID":"1231f0e6-4e20-4f34-b004-b7a284cf26d8","Type":"ContainerStarted","Data":"f219db3517abd3ba7f777b597d86586a902f75f7244d4eba083c7da27a3a1c83"} Oct 30 13:05:34.300390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:34.300342 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" podStartSLOduration=2.300305081 podStartE2EDuration="2.300305081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:34.299982348 +0000 UTC m=+841.115176771" watchObservedRunningTime="2026-10-30 13:05:34.300305081 +0000 UTC m=+841.115499504" Oct 30 13:05:34.627720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:34.627630 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:35.262966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:35.262933 5049 generic.go:356] "Generic (PLEG): container finished" podID="1231f0e6-4e20-4f34-b004-b7a284cf26d8" containerID="d363d56ac9bd33377bd2f213f410dfb36bcaea11744554ad8a4ed78334547afc" exitCode=0 Oct 30 13:05:35.263178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:35.263013 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" event={"ID":"1231f0e6-4e20-4f34-b004-b7a284cf26d8","Type":"ContainerDied","Data":"d363d56ac9bd33377bd2f213f410dfb36bcaea11744554ad8a4ed78334547afc"} Oct 30 13:05:36.510395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.510111 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:36.570819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.570769 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1231f0e6-4e20-4f34-b004-b7a284cf26d8-kube-api-access-s2pbs\" (UniqueName: \"kubernetes.io/projected/1231f0e6-4e20-4f34-b004-b7a284cf26d8-kube-api-access-s2pbs\") pod \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " Oct 30 13:05:36.570819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.570820 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-openshift-ca\") pod \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " Oct 30 13:05:36.571085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.570968 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-xks-ca\") pod \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\" (UID: \"1231f0e6-4e20-4f34-b004-b7a284cf26d8\") " Oct 30 13:05:36.571548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.571517 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-openshift-ca" pod "1231f0e6-4e20-4f34-b004-b7a284cf26d8" (UID: "1231f0e6-4e20-4f34-b004-b7a284cf26d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:36.574656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.574613 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-xks-ca" pod "1231f0e6-4e20-4f34-b004-b7a284cf26d8" (UID: "1231f0e6-4e20-4f34-b004-b7a284cf26d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:36.574766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.574692 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1231f0e6-4e20-4f34-b004-b7a284cf26d8-kube-api-access-s2pbs" pod "1231f0e6-4e20-4f34-b004-b7a284cf26d8" (UID: "1231f0e6-4e20-4f34-b004-b7a284cf26d8"). InnerVolumeSpecName "kube-api-access-s2pbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:36.683345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.672400 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2pbs\" (UniqueName: \"kubernetes.io/projected/1231f0e6-4e20-4f34-b004-b7a284cf26d8-kube-api-access-s2pbs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:36.683345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.672436 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:36.683345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.672452 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1231f0e6-4e20-4f34-b004-b7a284cf26d8-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:36.735795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:36.735755 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:37.283477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:37.283443 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" event={"ID":"1231f0e6-4e20-4f34-b004-b7a284cf26d8","Type":"ContainerDied","Data":"f219db3517abd3ba7f777b597d86586a902f75f7244d4eba083c7da27a3a1c83"} Oct 30 13:05:37.283477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:37.283458 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx" Oct 30 13:05:37.283756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:37.283495 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f219db3517abd3ba7f777b597d86586a902f75f7244d4eba083c7da27a3a1c83" Oct 30 13:05:38.593904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:38.593872 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:39.347588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.347547 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8"] Oct 30 13:05:39.348143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.348125 5049 cpu_manager.go:414] "containerMap: removing container" podUID="aec47a3e-9af4-4396-bc97-85c8ae728b92" containerName="revoke-keys" Oct 30 13:05:39.348244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.348145 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec47a3e-9af4-4396-bc97-85c8ae728b92" containerName="revoke-keys" Oct 30 13:05:39.348244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.348179 5049 cpu_manager.go:414] "containerMap: removing container" podUID="1231f0e6-4e20-4f34-b004-b7a284cf26d8" containerName="revoke-keys" Oct 30 13:05:39.348244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.348187 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1231f0e6-4e20-4f34-b004-b7a284cf26d8" containerName="revoke-keys" Oct 30 13:05:39.348452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.348300 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec47a3e-9af4-4396-bc97-85c8ae728b92" containerName="revoke-keys" Oct 30 13:05:39.348452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.348316 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="1231f0e6-4e20-4f34-b004-b7a284cf26d8" containerName="revoke-keys" Oct 30 13:05:39.351864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.351845 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:39.382669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.381491 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:05:39.382669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.381542 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:39.382669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.382063 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8"] Oct 30 13:05:39.424719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.424691 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ae9417-p8zq8\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:39.424878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.424836 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v4x9\" (UniqueName: \"kubernetes.io/projected/6721e397-4f87-450a-b39f-502c4bb65e9b-kube-api-access-9v4x9\") pod \"maas-api-revoke-sub-01ae9417-p8zq8\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:39.424950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.424907 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ae9417-p8zq8\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:39.526116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.526086 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ae9417-p8zq8\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:39.526309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.526180 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v4x9\" (UniqueName: \"kubernetes.io/projected/6721e397-4f87-450a-b39f-502c4bb65e9b-kube-api-access-9v4x9\") pod \"maas-api-revoke-sub-01ae9417-p8zq8\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:39.526309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.526232 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ae9417-p8zq8\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:39.527033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.527000 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ae9417-p8zq8\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:39.528671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.528648 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ae9417-p8zq8\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:39.548169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.548142 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v4x9\" (UniqueName: \"kubernetes.io/projected/6721e397-4f87-450a-b39f-502c4bb65e9b-kube-api-access-9v4x9\") pod \"maas-api-revoke-sub-01ae9417-p8zq8\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:39.693019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:39.692940 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:40.002434 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:40.002404 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8"] Oct 30 13:05:40.004098 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:40.004072 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6721e397_4f87_450a_b39f_502c4bb65e9b.slice/crio-5482a1954ed7839790b6d94a88893b3d944743ec50474c0df5699864439aff65 WatchSource:0}: Error finding container 5482a1954ed7839790b6d94a88893b3d944743ec50474c0df5699864439aff65: Status 404 returned error can't find the container with id 5482a1954ed7839790b6d94a88893b3d944743ec50474c0df5699864439aff65 Oct 30 13:05:40.302495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:40.302457 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" event={"ID":"6721e397-4f87-450a-b39f-502c4bb65e9b","Type":"ContainerStarted","Data":"5482a1954ed7839790b6d94a88893b3d944743ec50474c0df5699864439aff65"} Oct 30 13:05:40.597048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:40.596965 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:41.309553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:41.309518 5049 generic.go:356] "Generic (PLEG): container finished" podID="6721e397-4f87-450a-b39f-502c4bb65e9b" containerID="cbc9d757ad24c8f35138f70181251d0f2f0f665fcbec422563a037f288c08a24" exitCode=0 Oct 30 13:05:41.309966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:41.309583 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" event={"ID":"6721e397-4f87-450a-b39f-502c4bb65e9b","Type":"ContainerDied","Data":"cbc9d757ad24c8f35138f70181251d0f2f0f665fcbec422563a037f288c08a24"} Oct 30 13:05:42.531230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.531151 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:42.559447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.559410 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6721e397-4f87-450a-b39f-502c4bb65e9b-kube-api-access-9v4x9\" (UniqueName: \"kubernetes.io/projected/6721e397-4f87-450a-b39f-502c4bb65e9b-kube-api-access-9v4x9\") pod \"6721e397-4f87-450a-b39f-502c4bb65e9b\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " Oct 30 13:05:42.560057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.560034 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-xks-ca\") pod \"6721e397-4f87-450a-b39f-502c4bb65e9b\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " Oct 30 13:05:42.560241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.560222 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-openshift-ca\") pod \"6721e397-4f87-450a-b39f-502c4bb65e9b\" (UID: \"6721e397-4f87-450a-b39f-502c4bb65e9b\") " Oct 30 13:05:42.561399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.560875 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-openshift-ca" pod "6721e397-4f87-450a-b39f-502c4bb65e9b" (UID: "6721e397-4f87-450a-b39f-502c4bb65e9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:42.577937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.577856 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:42.579449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.579414 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6721e397-4f87-450a-b39f-502c4bb65e9b-kube-api-access-9v4x9" pod "6721e397-4f87-450a-b39f-502c4bb65e9b" (UID: "6721e397-4f87-450a-b39f-502c4bb65e9b"). InnerVolumeSpecName "kube-api-access-9v4x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:42.583098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.583062 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-xks-ca" pod "6721e397-4f87-450a-b39f-502c4bb65e9b" (UID: "6721e397-4f87-450a-b39f-502c4bb65e9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:42.661281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.661248 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:42.661281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.661276 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v4x9\" (UniqueName: \"kubernetes.io/projected/6721e397-4f87-450a-b39f-502c4bb65e9b-kube-api-access-9v4x9\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:42.661281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.661287 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6721e397-4f87-450a-b39f-502c4bb65e9b-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:42.932446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.932358 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9"] Oct 30 13:05:42.932976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.932957 5049 cpu_manager.go:414] "containerMap: removing container" podUID="6721e397-4f87-450a-b39f-502c4bb65e9b" containerName="revoke-keys" Oct 30 13:05:42.933077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.932979 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6721e397-4f87-450a-b39f-502c4bb65e9b" containerName="revoke-keys" Oct 30 13:05:42.933149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.933135 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="6721e397-4f87-450a-b39f-502c4bb65e9b" containerName="revoke-keys" Oct 30 13:05:42.936101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.936081 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:42.958501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.958470 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9"] Oct 30 13:05:42.965608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.965576 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2d2\" (UniqueName: \"kubernetes.io/projected/31fc3bef-c3c7-4820-8118-c76a37e8df47-kube-api-access-cc2d2\") pod \"maas-api-revoke-sub-5dea114b-dd2s9\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:42.965771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.965639 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea114b-dd2s9\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:42.965841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:42.965773 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea114b-dd2s9\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:43.066561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.066527 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea114b-dd2s9\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:43.066756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.066615 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2d2\" (UniqueName: \"kubernetes.io/projected/31fc3bef-c3c7-4820-8118-c76a37e8df47-kube-api-access-cc2d2\") pod \"maas-api-revoke-sub-5dea114b-dd2s9\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:43.066756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.066645 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea114b-dd2s9\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:43.067262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.067237 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea114b-dd2s9\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:43.068919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.068898 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea114b-dd2s9\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:43.088724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.088702 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2d2\" (UniqueName: \"kubernetes.io/projected/31fc3bef-c3c7-4820-8118-c76a37e8df47-kube-api-access-cc2d2\") pod \"maas-api-revoke-sub-5dea114b-dd2s9\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:43.249713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.249638 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:43.331638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.331609 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" Oct 30 13:05:43.336898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.336842 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8" event={"ID":"6721e397-4f87-450a-b39f-502c4bb65e9b","Type":"ContainerDied","Data":"5482a1954ed7839790b6d94a88893b3d944743ec50474c0df5699864439aff65"} Oct 30 13:05:43.336898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.336897 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5482a1954ed7839790b6d94a88893b3d944743ec50474c0df5699864439aff65" Oct 30 13:05:43.559769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:43.559735 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9"] Oct 30 13:05:43.564397 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:43.564372 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fc3bef_c3c7_4820_8118_c76a37e8df47.slice/crio-eda32dbbf786b9a92c7883bef344f309329ed124651829a21c2ff17b0f34ab84 WatchSource:0}: Error finding container eda32dbbf786b9a92c7883bef344f309329ed124651829a21c2ff17b0f34ab84: Status 404 returned error can't find the container with id eda32dbbf786b9a92c7883bef344f309329ed124651829a21c2ff17b0f34ab84 Oct 30 13:05:44.340882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.340848 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" event={"ID":"31fc3bef-c3c7-4820-8118-c76a37e8df47","Type":"ContainerStarted","Data":"02a136e1088ada466f26074e6063ed142a99bf56529b7c862b398ca5e9f4df40"} Oct 30 13:05:44.341077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.340899 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" event={"ID":"31fc3bef-c3c7-4820-8118-c76a37e8df47","Type":"ContainerStarted","Data":"eda32dbbf786b9a92c7883bef344f309329ed124651829a21c2ff17b0f34ab84"} Oct 30 13:05:44.391168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.391119 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" podStartSLOduration=2.391106247 podStartE2EDuration="2.391106247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:44.389704849 +0000 UTC m=+851.204899272" watchObservedRunningTime="2026-10-30 13:05:44.391106247 +0000 UTC m=+851.206300671" Oct 30 13:05:44.445637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.445601 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:44.582197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.582162 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg"] Oct 30 13:05:44.586593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.586573 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:44.608836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.608807 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg"] Oct 30 13:05:44.694051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.694014 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5350dd6a-69ckg\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:44.694246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.694087 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5350dd6a-69ckg\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:44.694246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.694211 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ntv\" (UniqueName: \"kubernetes.io/projected/2b780527-09c3-41ff-b3c6-58f78aae38ac-kube-api-access-87ntv\") pod \"maas-api-revoke-sub-5350dd6a-69ckg\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:44.795201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.795166 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ntv\" (UniqueName: \"kubernetes.io/projected/2b780527-09c3-41ff-b3c6-58f78aae38ac-kube-api-access-87ntv\") pod \"maas-api-revoke-sub-5350dd6a-69ckg\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:44.795364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.795348 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5350dd6a-69ckg\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:44.795417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.795401 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5350dd6a-69ckg\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:44.796288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.796263 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5350dd6a-69ckg\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:44.797867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.797843 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5350dd6a-69ckg\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:44.816740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.816715 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ntv\" (UniqueName: \"kubernetes.io/projected/2b780527-09c3-41ff-b3c6-58f78aae38ac-kube-api-access-87ntv\") pod \"maas-api-revoke-sub-5350dd6a-69ckg\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:44.900894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:44.900799 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:45.202434 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:45.202405 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg"] Oct 30 13:05:45.202974 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:45.202941 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b780527_09c3_41ff_b3c6_58f78aae38ac.slice/crio-8dcb9aacbc1097560c3de909e447d6f63f8dbd798fe81ffd93687f8150a71b71 WatchSource:0}: Error finding container 8dcb9aacbc1097560c3de909e447d6f63f8dbd798fe81ffd93687f8150a71b71: Status 404 returned error can't find the container with id 8dcb9aacbc1097560c3de909e447d6f63f8dbd798fe81ffd93687f8150a71b71 Oct 30 13:05:45.348377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:45.348338 5049 generic.go:356] "Generic (PLEG): container finished" podID="31fc3bef-c3c7-4820-8118-c76a37e8df47" containerID="02a136e1088ada466f26074e6063ed142a99bf56529b7c862b398ca5e9f4df40" exitCode=0 Oct 30 13:05:45.348560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:45.348400 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" event={"ID":"31fc3bef-c3c7-4820-8118-c76a37e8df47","Type":"ContainerDied","Data":"02a136e1088ada466f26074e6063ed142a99bf56529b7c862b398ca5e9f4df40"} Oct 30 13:05:45.349788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:45.349767 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" event={"ID":"2b780527-09c3-41ff-b3c6-58f78aae38ac","Type":"ContainerStarted","Data":"8dcb9aacbc1097560c3de909e447d6f63f8dbd798fe81ffd93687f8150a71b71"} Oct 30 13:05:46.354666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.354629 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:46.357440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.357406 5049 generic.go:356] "Generic (PLEG): container finished" podID="2b780527-09c3-41ff-b3c6-58f78aae38ac" containerID="e936f0265a3c97df6758401097cd922f4242bc823e81bdae743e53071e8d02d8" exitCode=0 Oct 30 13:05:46.357563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.357487 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" event={"ID":"2b780527-09c3-41ff-b3c6-58f78aae38ac","Type":"ContainerDied","Data":"e936f0265a3c97df6758401097cd922f4242bc823e81bdae743e53071e8d02d8"} Oct 30 13:05:46.502384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.502361 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:46.616445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.616411 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31fc3bef-c3c7-4820-8118-c76a37e8df47-kube-api-access-cc2d2\" (UniqueName: \"kubernetes.io/projected/31fc3bef-c3c7-4820-8118-c76a37e8df47-kube-api-access-cc2d2\") pod \"31fc3bef-c3c7-4820-8118-c76a37e8df47\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " Oct 30 13:05:46.616445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.616448 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-openshift-ca\") pod \"31fc3bef-c3c7-4820-8118-c76a37e8df47\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " Oct 30 13:05:46.616654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.616486 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-xks-ca\") pod \"31fc3bef-c3c7-4820-8118-c76a37e8df47\" (UID: \"31fc3bef-c3c7-4820-8118-c76a37e8df47\") " Oct 30 13:05:46.616831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.616807 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-openshift-ca" pod "31fc3bef-c3c7-4820-8118-c76a37e8df47" (UID: "31fc3bef-c3c7-4820-8118-c76a37e8df47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:46.618567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.618541 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-xks-ca" pod "31fc3bef-c3c7-4820-8118-c76a37e8df47" (UID: "31fc3bef-c3c7-4820-8118-c76a37e8df47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:46.618729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.618706 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fc3bef-c3c7-4820-8118-c76a37e8df47-kube-api-access-cc2d2" pod "31fc3bef-c3c7-4820-8118-c76a37e8df47" (UID: "31fc3bef-c3c7-4820-8118-c76a37e8df47"). InnerVolumeSpecName "kube-api-access-cc2d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:46.717932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.717850 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc2d2\" (UniqueName: \"kubernetes.io/projected/31fc3bef-c3c7-4820-8118-c76a37e8df47-kube-api-access-cc2d2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:46.717932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.717880 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:46.717932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:46.717890 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fc3bef-c3c7-4820-8118-c76a37e8df47-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.372748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.372719 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" Oct 30 13:05:47.373546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.372735 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9" event={"ID":"31fc3bef-c3c7-4820-8118-c76a37e8df47","Type":"ContainerDied","Data":"eda32dbbf786b9a92c7883bef344f309329ed124651829a21c2ff17b0f34ab84"} Oct 30 13:05:47.373546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.373476 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eda32dbbf786b9a92c7883bef344f309329ed124651829a21c2ff17b0f34ab84" Oct 30 13:05:47.544684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.544659 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:47.628782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.628688 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-openshift-ca\") pod \"2b780527-09c3-41ff-b3c6-58f78aae38ac\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " Oct 30 13:05:47.628782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.628754 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-xks-ca\") pod \"2b780527-09c3-41ff-b3c6-58f78aae38ac\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " Oct 30 13:05:47.629020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.628873 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b780527-09c3-41ff-b3c6-58f78aae38ac-kube-api-access-87ntv\" (UniqueName: \"kubernetes.io/projected/2b780527-09c3-41ff-b3c6-58f78aae38ac-kube-api-access-87ntv\") pod \"2b780527-09c3-41ff-b3c6-58f78aae38ac\" (UID: \"2b780527-09c3-41ff-b3c6-58f78aae38ac\") " Oct 30 13:05:47.629146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.629115 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-openshift-ca" pod "2b780527-09c3-41ff-b3c6-58f78aae38ac" (UID: "2b780527-09c3-41ff-b3c6-58f78aae38ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:47.629351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.629333 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.630979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.630952 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-xks-ca" pod "2b780527-09c3-41ff-b3c6-58f78aae38ac" (UID: "2b780527-09c3-41ff-b3c6-58f78aae38ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:47.631303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.631281 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b780527-09c3-41ff-b3c6-58f78aae38ac-kube-api-access-87ntv" pod "2b780527-09c3-41ff-b3c6-58f78aae38ac" (UID: "2b780527-09c3-41ff-b3c6-58f78aae38ac"). InnerVolumeSpecName "kube-api-access-87ntv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:47.730173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.730141 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b780527-09c3-41ff-b3c6-58f78aae38ac-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.730173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.730167 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ntv\" (UniqueName: \"kubernetes.io/projected/2b780527-09c3-41ff-b3c6-58f78aae38ac-kube-api-access-87ntv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.817563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.817528 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v"] Oct 30 13:05:47.818116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.818100 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2b780527-09c3-41ff-b3c6-58f78aae38ac" containerName="revoke-keys" Oct 30 13:05:47.818171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.818119 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b780527-09c3-41ff-b3c6-58f78aae38ac" containerName="revoke-keys" Oct 30 13:05:47.818171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.818133 5049 cpu_manager.go:414] "containerMap: removing container" podUID="31fc3bef-c3c7-4820-8118-c76a37e8df47" containerName="revoke-keys" Oct 30 13:05:47.818171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.818138 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31fc3bef-c3c7-4820-8118-c76a37e8df47" containerName="revoke-keys" Oct 30 13:05:47.818261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.818237 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b780527-09c3-41ff-b3c6-58f78aae38ac" containerName="revoke-keys" Oct 30 13:05:47.818261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.818246 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="31fc3bef-c3c7-4820-8118-c76a37e8df47" containerName="revoke-keys" Oct 30 13:05:47.822008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.821987 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:47.844244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.844196 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v"] Oct 30 13:05:47.933257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.933161 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c097f0d-8sm6v\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:47.933257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.933209 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c097f0d-8sm6v\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:47.933257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:47.933232 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjwp\" (UniqueName: \"kubernetes.io/projected/d9f5817b-e848-4027-9de6-8414a4cb3a91-kube-api-access-4bjwp\") pod \"maas-api-revoke-sub-2c097f0d-8sm6v\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:48.034479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.034440 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c097f0d-8sm6v\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:48.034652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.034495 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c097f0d-8sm6v\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:48.034652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.034530 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjwp\" (UniqueName: \"kubernetes.io/projected/d9f5817b-e848-4027-9de6-8414a4cb3a91-kube-api-access-4bjwp\") pod \"maas-api-revoke-sub-2c097f0d-8sm6v\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:48.035162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.035139 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c097f0d-8sm6v\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:48.037030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.037003 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c097f0d-8sm6v\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:48.056073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.056048 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjwp\" (UniqueName: \"kubernetes.io/projected/d9f5817b-e848-4027-9de6-8414a4cb3a91-kube-api-access-4bjwp\") pod \"maas-api-revoke-sub-2c097f0d-8sm6v\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:48.127475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.122475 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:48.138277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.138243 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:48.379490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.379460 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" Oct 30 13:05:48.379490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.379482 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg" event={"ID":"2b780527-09c3-41ff-b3c6-58f78aae38ac","Type":"ContainerDied","Data":"8dcb9aacbc1097560c3de909e447d6f63f8dbd798fe81ffd93687f8150a71b71"} Oct 30 13:05:48.380035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.379547 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dcb9aacbc1097560c3de909e447d6f63f8dbd798fe81ffd93687f8150a71b71" Oct 30 13:05:48.477110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:48.477084 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v"] Oct 30 13:05:48.479832 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:48.479804 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f5817b_e848_4027_9de6_8414a4cb3a91.slice/crio-f3913447c0f8ba09a79560a54178ca2411ff12dc47e22fc94fb44d533a9af743 WatchSource:0}: Error finding container f3913447c0f8ba09a79560a54178ca2411ff12dc47e22fc94fb44d533a9af743: Status 404 returned error can't find the container with id f3913447c0f8ba09a79560a54178ca2411ff12dc47e22fc94fb44d533a9af743 Oct 30 13:05:49.387041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:49.387007 5049 generic.go:356] "Generic (PLEG): container finished" podID="d9f5817b-e848-4027-9de6-8414a4cb3a91" containerID="f0688171d6efe6017cdd18bbc44281736e278095587fb1961a159acdea9ff81e" exitCode=0 Oct 30 13:05:49.387576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:49.387089 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" event={"ID":"d9f5817b-e848-4027-9de6-8414a4cb3a91","Type":"ContainerDied","Data":"f0688171d6efe6017cdd18bbc44281736e278095587fb1961a159acdea9ff81e"} Oct 30 13:05:49.387576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:49.387140 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" event={"ID":"d9f5817b-e848-4027-9de6-8414a4cb3a91","Type":"ContainerStarted","Data":"f3913447c0f8ba09a79560a54178ca2411ff12dc47e22fc94fb44d533a9af743"} Oct 30 13:05:50.094155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.094117 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:50.565250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.565226 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:50.675217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.675120 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-xks-ca\") pod \"d9f5817b-e848-4027-9de6-8414a4cb3a91\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " Oct 30 13:05:50.675399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.675229 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-openshift-ca\") pod \"d9f5817b-e848-4027-9de6-8414a4cb3a91\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " Oct 30 13:05:50.675399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.675394 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f5817b-e848-4027-9de6-8414a4cb3a91-kube-api-access-4bjwp\" (UniqueName: \"kubernetes.io/projected/d9f5817b-e848-4027-9de6-8414a4cb3a91-kube-api-access-4bjwp\") pod \"d9f5817b-e848-4027-9de6-8414a4cb3a91\" (UID: \"d9f5817b-e848-4027-9de6-8414a4cb3a91\") " Oct 30 13:05:50.676020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.675688 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-openshift-ca" pod "d9f5817b-e848-4027-9de6-8414a4cb3a91" (UID: "d9f5817b-e848-4027-9de6-8414a4cb3a91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:50.676020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.675840 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:50.677722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.677694 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-xks-ca" pod "d9f5817b-e848-4027-9de6-8414a4cb3a91" (UID: "d9f5817b-e848-4027-9de6-8414a4cb3a91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:50.678863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.678839 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f5817b-e848-4027-9de6-8414a4cb3a91-kube-api-access-4bjwp" pod "d9f5817b-e848-4027-9de6-8414a4cb3a91" (UID: "d9f5817b-e848-4027-9de6-8414a4cb3a91"). InnerVolumeSpecName "kube-api-access-4bjwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:50.777122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.777087 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bjwp\" (UniqueName: \"kubernetes.io/projected/d9f5817b-e848-4027-9de6-8414a4cb3a91-kube-api-access-4bjwp\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:50.777122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:50.777118 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f5817b-e848-4027-9de6-8414a4cb3a91-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:51.412358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:51.412278 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" event={"ID":"d9f5817b-e848-4027-9de6-8414a4cb3a91","Type":"ContainerDied","Data":"f3913447c0f8ba09a79560a54178ca2411ff12dc47e22fc94fb44d533a9af743"} Oct 30 13:05:51.412358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:51.412360 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3913447c0f8ba09a79560a54178ca2411ff12dc47e22fc94fb44d533a9af743" Oct 30 13:05:51.412616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:51.412435 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v" Oct 30 13:05:52.160596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.160557 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:52.211572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.211528 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7"] Oct 30 13:05:52.212175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.212155 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d9f5817b-e848-4027-9de6-8414a4cb3a91" containerName="revoke-keys" Oct 30 13:05:52.212175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.212175 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f5817b-e848-4027-9de6-8414a4cb3a91" containerName="revoke-keys" Oct 30 13:05:52.212394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.212380 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f5817b-e848-4027-9de6-8414a4cb3a91" containerName="revoke-keys" Oct 30 13:05:52.216083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.216061 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.233247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.233226 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:52.244003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.243981 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:05:52.248352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.248313 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7"] Oct 30 13:05:52.297669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.297622 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-addee7da-5njc7\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.297842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.297736 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s7fh\" (UniqueName: \"kubernetes.io/projected/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-kube-api-access-8s7fh\") pod \"maas-api-revoke-sub-addee7da-5njc7\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.297842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.297825 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-addee7da-5njc7\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.398710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.398671 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-addee7da-5njc7\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.398890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.398798 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-addee7da-5njc7\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.398890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.398819 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s7fh\" (UniqueName: \"kubernetes.io/projected/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-kube-api-access-8s7fh\") pod \"maas-api-revoke-sub-addee7da-5njc7\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.399463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.399437 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-addee7da-5njc7\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.401269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.401248 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-addee7da-5njc7\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.428489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.428414 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s7fh\" (UniqueName: \"kubernetes.io/projected/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-kube-api-access-8s7fh\") pod \"maas-api-revoke-sub-addee7da-5njc7\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.531181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.531144 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:52.820479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.820444 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v"] Oct 30 13:05:52.826485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.826459 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:52.841215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.841183 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cb58b8-openshift-default-dockercfg-dftld\"" Oct 30 13:05:52.857702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.857666 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v"] Oct 30 13:05:52.862060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.862032 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7"] Oct 30 13:05:52.907732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.907680 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:52.907890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.907761 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:52.907942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.907903 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:52.907984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.907961 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjlc\" (UniqueName: \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-kube-api-access-6hjlc\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:52.908135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.908111 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:52.908197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.908152 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:52.908197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.908180 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:52.908274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.908261 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:52.908350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:52.908334 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.009149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.009106 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.009359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.009180 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.009359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.009254 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.009646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.009575 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.009646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.009313 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjlc\" (UniqueName: \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-kube-api-access-6hjlc\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.009847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.009697 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.009847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.009727 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.009847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.009753 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.009847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.009810 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.010104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.009960 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.010104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.010031 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.010178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.010156 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.010243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.010223 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.010286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.010242 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.011891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.011861 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.012643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.012618 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.030881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.030846 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjlc\" (UniqueName: \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-kube-api-access-6hjlc\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.032588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.032566 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.144000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.143964 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:53.171690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.171660 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4"] Oct 30 13:05:53.176806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.176786 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.193103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.193081 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e34fd5-openshift-default-dockercfg-mz9s9\"" Oct 30 13:05:53.229821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.229796 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4"] Oct 30 13:05:53.324804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.324759 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.324988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.324904 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.325042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.324987 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.325042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.325024 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.325118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.325089 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.325118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.325113 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.325211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.325145 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745de12-8e89-4082-9789-262034268d63-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.325282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.325267 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745de12-8e89-4082-9789-262034268d63-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.325330 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.325295 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nbr\" (UniqueName: \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-kube-api-access-h8nbr\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426225 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nbr\" (UniqueName: \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-kube-api-access-h8nbr\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426372 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426484 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426524 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426557 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426639 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426674 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426722 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745de12-8e89-4082-9789-262034268d63-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426808 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745de12-8e89-4082-9789-262034268d63-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426940 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.426965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.426956 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.427353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.427285 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.427543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.427520 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.427831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.427806 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745de12-8e89-4082-9789-262034268d63-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.429269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.429244 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.429835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.429814 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745de12-8e89-4082-9789-262034268d63-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.433622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.433596 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" event={"ID":"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026","Type":"ContainerStarted","Data":"54342181d6a93f22d63c90934db286db6fff25c307d0282f74b4d6604a8aa1fb"} Oct 30 13:05:53.433709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.433652 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" event={"ID":"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026","Type":"ContainerStarted","Data":"99a4dab1315b2854d7456493a718e9c889bbef74f046c73e330d68eecd4b5910"} Oct 30 13:05:53.448315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.448291 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nbr\" (UniqueName: \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-kube-api-access-h8nbr\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.449837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.449813 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.467862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.467839 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v"] Oct 30 13:05:53.469334 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:53.469282 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa99dc8_e3a2_4395_a917_c6b1f6e5d273.slice/crio-3f564d093fe4a573b8a59742bfae2b27cb94712c308f47eb27774061f6ad8a53 WatchSource:0}: Error finding container 3f564d093fe4a573b8a59742bfae2b27cb94712c308f47eb27774061f6ad8a53: Status 404 returned error can't find the container with id 3f564d093fe4a573b8a59742bfae2b27cb94712c308f47eb27774061f6ad8a53 Oct 30 13:05:53.471296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.471259 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" podStartSLOduration=1.471246497 podStartE2EDuration="1.471246497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:53.470026141 +0000 UTC m=+860.285220563" watchObservedRunningTime="2026-10-30 13:05:53.471246497 +0000 UTC m=+860.286440936" Oct 30 13:05:53.471939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.471906 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:53.472005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.471975 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:53.472044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.472008 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:53.493647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.493627 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:53.822805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.822756 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4"] Oct 30 13:05:53.826610 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:53.826573 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf745de12_8e89_4082_9789_262034268d63.slice/crio-450719cdfee8a6df5dccb7be8d27407ac6e04a5deaa22ab44e80c0b3b0a2c6a8 WatchSource:0}: Error finding container 450719cdfee8a6df5dccb7be8d27407ac6e04a5deaa22ab44e80c0b3b0a2c6a8: Status 404 returned error can't find the container with id 450719cdfee8a6df5dccb7be8d27407ac6e04a5deaa22ab44e80c0b3b0a2c6a8 Oct 30 13:05:53.830491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.830449 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:53.830610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.830541 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:53.830610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:53.830586 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:54.100301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.100200 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:54.448724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.448688 5049 generic.go:356] "Generic (PLEG): container finished" podID="2dcc4b39-15f3-4c12-bc46-56c3ec7d4026" containerID="54342181d6a93f22d63c90934db286db6fff25c307d0282f74b4d6604a8aa1fb" exitCode=0 Oct 30 13:05:54.449187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.448797 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" event={"ID":"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026","Type":"ContainerDied","Data":"54342181d6a93f22d63c90934db286db6fff25c307d0282f74b4d6604a8aa1fb"} Oct 30 13:05:54.450626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.450604 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" event={"ID":"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273","Type":"ContainerStarted","Data":"337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887"} Oct 30 13:05:54.450837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.450822 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" event={"ID":"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273","Type":"ContainerStarted","Data":"3f564d093fe4a573b8a59742bfae2b27cb94712c308f47eb27774061f6ad8a53"} Oct 30 13:05:54.452762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.452736 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" event={"ID":"f745de12-8e89-4082-9789-262034268d63","Type":"ContainerStarted","Data":"6079ce6bcb8476e702e7287127384a27ab56e63c8b6b4654899a11635540cecf"} Oct 30 13:05:54.452880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.452786 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" event={"ID":"f745de12-8e89-4082-9789-262034268d63","Type":"ContainerStarted","Data":"450719cdfee8a6df5dccb7be8d27407ac6e04a5deaa22ab44e80c0b3b0a2c6a8"} Oct 30 13:05:54.494618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.494586 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:05:54.496182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.496156 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:05:54.496293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.496199 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:05:54.581291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.581241 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" podStartSLOduration=2.581226642 podStartE2EDuration="2.581226642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:54.53978371 +0000 UTC m=+861.354978137" watchObservedRunningTime="2026-10-30 13:05:54.581226642 +0000 UTC m=+861.396421106" Oct 30 13:05:54.583072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:54.583025 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" podStartSLOduration=1.583008966 podStartE2EDuration="1.583008966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:54.580732975 +0000 UTC m=+861.395927399" watchObservedRunningTime="2026-10-30 13:05:54.583008966 +0000 UTC m=+861.398203390" Oct 30 13:05:55.158800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.158708 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:05:55.163823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.163477 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:05:55.163823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.163538 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:05:55.494063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.494035 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:05:55.494509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.494485 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:05:55.733599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.733518 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:55.879038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.879001 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-openshift-ca\") pod \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " Oct 30 13:05:55.879217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.879147 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-kube-api-access-8s7fh\" (UniqueName: \"kubernetes.io/projected/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-kube-api-access-8s7fh\") pod \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " Oct 30 13:05:55.879217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.879176 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-xks-ca\") pod \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\" (UID: \"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026\") " Oct 30 13:05:55.879429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.879402 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-openshift-ca" pod "2dcc4b39-15f3-4c12-bc46-56c3ec7d4026" (UID: "2dcc4b39-15f3-4c12-bc46-56c3ec7d4026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:55.879579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.879561 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:55.881215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.881187 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-kube-api-access-8s7fh" pod "2dcc4b39-15f3-4c12-bc46-56c3ec7d4026" (UID: "2dcc4b39-15f3-4c12-bc46-56c3ec7d4026"). InnerVolumeSpecName "kube-api-access-8s7fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:55.881479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.881460 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-xks-ca" pod "2dcc4b39-15f3-4c12-bc46-56c3ec7d4026" (UID: "2dcc4b39-15f3-4c12-bc46-56c3ec7d4026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:55.980455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.980418 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s7fh\" (UniqueName: \"kubernetes.io/projected/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-kube-api-access-8s7fh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:55.980455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:55.980443 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.047405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:56.041367 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:56.148357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:56.146712 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:05:56.148357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:56.146774 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:05:56.475300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:56.473595 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" Oct 30 13:05:56.475300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:56.473611 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7" event={"ID":"2dcc4b39-15f3-4c12-bc46-56c3ec7d4026","Type":"ContainerDied","Data":"99a4dab1315b2854d7456493a718e9c889bbef74f046c73e330d68eecd4b5910"} Oct 30 13:05:56.475300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:56.473658 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99a4dab1315b2854d7456493a718e9c889bbef74f046c73e330d68eecd4b5910" Oct 30 13:05:56.494800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:56.494774 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:05:56.495215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:56.494816 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:05:56.683791 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:05:56.683729 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:05:56.957345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:56.953746 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:57.145341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:57.144732 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:05:57.145341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:57.144813 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:05:57.147744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:57.147657 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:57.494724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:57.494692 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:05:57.494920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:57.494753 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:05:58.079345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.078961 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:58.149344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.148549 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:05:58.149344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.148614 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:05:58.265343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.253365 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565"] Oct 30 13:05:58.265343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.254175 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2dcc4b39-15f3-4c12-bc46-56c3ec7d4026" containerName="revoke-keys" Oct 30 13:05:58.265343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.254198 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dcc4b39-15f3-4c12-bc46-56c3ec7d4026" containerName="revoke-keys" Oct 30 13:05:58.265343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.254426 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dcc4b39-15f3-4c12-bc46-56c3ec7d4026" containerName="revoke-keys" Oct 30 13:05:58.277459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.275638 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565"] Oct 30 13:05:58.277459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.275757 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:58.300473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.300451 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:58.300610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.300464 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:05:58.353406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.330171 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3466c89d-w4565\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:58.353785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.353755 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3466c89d-w4565\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:58.353910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.353894 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrnq\" (UniqueName: \"kubernetes.io/projected/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-kube-api-access-jkrnq\") pod \"maas-api-revoke-sub-3466c89d-w4565\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:58.455067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.455020 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3466c89d-w4565\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:58.455885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.455859 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrnq\" (UniqueName: \"kubernetes.io/projected/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-kube-api-access-jkrnq\") pod \"maas-api-revoke-sub-3466c89d-w4565\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:58.463219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.463189 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3466c89d-w4565\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:58.464596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.464097 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3466c89d-w4565\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:58.465066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.465017 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3466c89d-w4565\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:58.486253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.486227 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrnq\" (UniqueName: \"kubernetes.io/projected/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-kube-api-access-jkrnq\") pod \"maas-api-revoke-sub-3466c89d-w4565\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:58.494222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.493913 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:05:58.494222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.493956 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:05:58.596895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:58.596773 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:05:59.059110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.058698 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565"] Oct 30 13:05:59.061847 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:59.061801 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91b9bfc_9b4f_4a4a_bc3c_299c51529401.slice/crio-636e3bd082e0290ea6b4a74b3c883a9fdd5bf371c363aa66c374017c5911cf83 WatchSource:0}: Error finding container 636e3bd082e0290ea6b4a74b3c883a9fdd5bf371c363aa66c374017c5911cf83: Status 404 returned error can't find the container with id 636e3bd082e0290ea6b4a74b3c883a9fdd5bf371c363aa66c374017c5911cf83 Oct 30 13:05:59.163842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.163803 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2"] Oct 30 13:05:59.164476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.164105 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:05:59.164476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.164176 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:05:59.170689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.170547 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.190404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.190376 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2"] Oct 30 13:05:59.282105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.282074 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxkd5\" (UniqueName: \"kubernetes.io/projected/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-kube-api-access-qxkd5\") pod \"maas-api-revoke-sub-54ef410a-xg7w2\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.282248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.282153 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54ef410a-xg7w2\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.282248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.282202 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54ef410a-xg7w2\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.383830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.383757 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxkd5\" (UniqueName: \"kubernetes.io/projected/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-kube-api-access-qxkd5\") pod \"maas-api-revoke-sub-54ef410a-xg7w2\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.383971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.383831 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54ef410a-xg7w2\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.383971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.383877 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54ef410a-xg7w2\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.384659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.384638 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54ef410a-xg7w2\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.386426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.386405 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54ef410a-xg7w2\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.415035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.415007 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxkd5\" (UniqueName: \"kubernetes.io/projected/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-kube-api-access-qxkd5\") pod \"maas-api-revoke-sub-54ef410a-xg7w2\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.489529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.489502 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:05:59.495495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.495466 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:05:59.495621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.495523 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:05:59.503855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.503828 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" event={"ID":"b91b9bfc-9b4f-4a4a-bc3c-299c51529401","Type":"ContainerStarted","Data":"ab0f4300808c44780c318abb2ab31cecbc42d5ab1688d46e28563b1a2eb7163a"} Oct 30 13:05:59.503954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.503884 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" event={"ID":"b91b9bfc-9b4f-4a4a-bc3c-299c51529401","Type":"ContainerStarted","Data":"636e3bd082e0290ea6b4a74b3c883a9fdd5bf371c363aa66c374017c5911cf83"} Oct 30 13:05:59.572590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.572519 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" podStartSLOduration=1.572491802 podStartE2EDuration="1.572491802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:59.571036259 +0000 UTC m=+866.386230683" watchObservedRunningTime="2026-10-30 13:05:59.572491802 +0000 UTC m=+866.387686228" Oct 30 13:05:59.937597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:05:59.937566 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2"] Oct 30 13:05:59.939335 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:05:59.939291 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acfffb1_ff93_4bb6_a30c_d27bbc1328b8.slice/crio-8efb25ac47bb8ed432d8842bf2f0fb79082f9180d13790d21eeb777d9980a370 WatchSource:0}: Error finding container 8efb25ac47bb8ed432d8842bf2f0fb79082f9180d13790d21eeb777d9980a370: Status 404 returned error can't find the container with id 8efb25ac47bb8ed432d8842bf2f0fb79082f9180d13790d21eeb777d9980a370 Oct 30 13:06:00.127344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:00.120863 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:00.144618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:00.144285 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:06:00.144618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:00.144363 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 13:06:00.494003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:00.493960 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:06:00.494184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:00.494022 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 13:06:00.516519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:00.516483 5049 generic.go:356] "Generic (PLEG): container finished" podID="b91b9bfc-9b4f-4a4a-bc3c-299c51529401" containerID="ab0f4300808c44780c318abb2ab31cecbc42d5ab1688d46e28563b1a2eb7163a" exitCode=0 Oct 30 13:06:00.516703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:00.516683 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" event={"ID":"b91b9bfc-9b4f-4a4a-bc3c-299c51529401","Type":"ContainerDied","Data":"ab0f4300808c44780c318abb2ab31cecbc42d5ab1688d46e28563b1a2eb7163a"} Oct 30 13:06:00.518919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:00.518882 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" event={"ID":"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8","Type":"ContainerStarted","Data":"4b3a2dc53eac303cfbf0d30279d5d699f0a9e6270ee212c84dcfa69eedf9c640"} Oct 30 13:06:00.518919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:00.518923 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" event={"ID":"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8","Type":"ContainerStarted","Data":"8efb25ac47bb8ed432d8842bf2f0fb79082f9180d13790d21eeb777d9980a370"} Oct 30 13:06:00.591111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:00.591051 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" podStartSLOduration=1.591033235 podStartE2EDuration="1.591033235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:00.590889019 +0000 UTC m=+867.406083442" watchObservedRunningTime="2026-10-30 13:06:00.591033235 +0000 UTC m=+867.406227660" Oct 30 13:06:01.148996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.148901 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:06:01.149405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.149369 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:06:01.150180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.150156 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:06:01.499370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.499278 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:06:01.499723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.499699 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:06:01.500548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.500527 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:06:01.527405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.527360 5049 generic.go:356] "Generic (PLEG): container finished" podID="2acfffb1-ff93-4bb6-a30c-d27bbc1328b8" containerID="4b3a2dc53eac303cfbf0d30279d5d699f0a9e6270ee212c84dcfa69eedf9c640" exitCode=0 Oct 30 13:06:01.527405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.527399 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" event={"ID":"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8","Type":"ContainerDied","Data":"4b3a2dc53eac303cfbf0d30279d5d699f0a9e6270ee212c84dcfa69eedf9c640"} Oct 30 13:06:01.699080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.699051 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:06:01.711451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.711423 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-openshift-ca\") pod \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " Oct 30 13:06:01.711643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.711627 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-xks-ca\") pod \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " Oct 30 13:06:01.711697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.711670 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-kube-api-access-jkrnq\" (UniqueName: \"kubernetes.io/projected/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-kube-api-access-jkrnq\") pod \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\" (UID: \"b91b9bfc-9b4f-4a4a-bc3c-299c51529401\") " Oct 30 13:06:01.711843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.711814 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-openshift-ca" pod "b91b9bfc-9b4f-4a4a-bc3c-299c51529401" (UID: "b91b9bfc-9b4f-4a4a-bc3c-299c51529401"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:01.712019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.712004 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:01.714001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.713975 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-kube-api-access-jkrnq" pod "b91b9bfc-9b4f-4a4a-bc3c-299c51529401" (UID: "b91b9bfc-9b4f-4a4a-bc3c-299c51529401"). InnerVolumeSpecName "kube-api-access-jkrnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:01.714196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.714176 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-xks-ca" pod "b91b9bfc-9b4f-4a4a-bc3c-299c51529401" (UID: "b91b9bfc-9b4f-4a4a-bc3c-299c51529401"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:01.813697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.813661 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:01.813858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.813713 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrnq\" (UniqueName: \"kubernetes.io/projected/b91b9bfc-9b4f-4a4a-bc3c-299c51529401-kube-api-access-jkrnq\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:01.815590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:01.815544 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:02.535729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.535696 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" Oct 30 13:06:02.536203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.535764 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565" event={"ID":"b91b9bfc-9b4f-4a4a-bc3c-299c51529401","Type":"ContainerDied","Data":"636e3bd082e0290ea6b4a74b3c883a9fdd5bf371c363aa66c374017c5911cf83"} Oct 30 13:06:02.536203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.535814 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="636e3bd082e0290ea6b4a74b3c883a9fdd5bf371c363aa66c374017c5911cf83" Oct 30 13:06:02.540309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.540278 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z"] Oct 30 13:06:02.541117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.541092 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b91b9bfc-9b4f-4a4a-bc3c-299c51529401" containerName="revoke-keys" Oct 30 13:06:02.541117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.541115 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91b9bfc-9b4f-4a4a-bc3c-299c51529401" containerName="revoke-keys" Oct 30 13:06:02.541250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.541235 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91b9bfc-9b4f-4a4a-bc3c-299c51529401" containerName="revoke-keys" Oct 30 13:06:02.546618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.545337 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.569726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.569692 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z"] Oct 30 13:06:02.624794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.623419 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfl98\" (UniqueName: \"kubernetes.io/projected/46c410dc-efe9-4661-aa8d-bc27ddd62c16-kube-api-access-gfl98\") pod \"maas-api-revoke-sub-3a334383-z8p8z\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.624794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.623594 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a334383-z8p8z\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.624794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.623674 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a334383-z8p8z\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.724500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.724469 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfl98\" (UniqueName: \"kubernetes.io/projected/46c410dc-efe9-4661-aa8d-bc27ddd62c16-kube-api-access-gfl98\") pod \"maas-api-revoke-sub-3a334383-z8p8z\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.724676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.724588 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a334383-z8p8z\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.724676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.724647 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a334383-z8p8z\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.725403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.725375 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a334383-z8p8z\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.727673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.727630 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a334383-z8p8z\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.731957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.731924 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:06:02.746606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.746583 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfl98\" (UniqueName: \"kubernetes.io/projected/46c410dc-efe9-4661-aa8d-bc27ddd62c16-kube-api-access-gfl98\") pod \"maas-api-revoke-sub-3a334383-z8p8z\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.825925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.825892 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-xks-ca\") pod \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " Oct 30 13:06:02.826120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.825934 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-kube-api-access-qxkd5\" (UniqueName: \"kubernetes.io/projected/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-kube-api-access-qxkd5\") pod \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " Oct 30 13:06:02.826120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.825978 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-openshift-ca\") pod \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\" (UID: \"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8\") " Oct 30 13:06:02.826464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.826434 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-openshift-ca" pod "2acfffb1-ff93-4bb6-a30c-d27bbc1328b8" (UID: "2acfffb1-ff93-4bb6-a30c-d27bbc1328b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:02.826578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.826531 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:02.828283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.828256 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-kube-api-access-qxkd5" pod "2acfffb1-ff93-4bb6-a30c-d27bbc1328b8" (UID: "2acfffb1-ff93-4bb6-a30c-d27bbc1328b8"). InnerVolumeSpecName "kube-api-access-qxkd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:02.828494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.828477 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-xks-ca" pod "2acfffb1-ff93-4bb6-a30c-d27bbc1328b8" (UID: "2acfffb1-ff93-4bb6-a30c-d27bbc1328b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:02.863653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.863625 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:02.934204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.934128 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:02.934204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:02.934169 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxkd5\" (UniqueName: \"kubernetes.io/projected/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8-kube-api-access-qxkd5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:03.168924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.168893 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z"] Oct 30 13:06:03.169742 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:06:03.169717 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c410dc_efe9_4661_aa8d_bc27ddd62c16.slice/crio-f947ee0ff506133f29a8d193fa40d428802dff1a65bc071679ff0ae64b6694e9 WatchSource:0}: Error finding container f947ee0ff506133f29a8d193fa40d428802dff1a65bc071679ff0ae64b6694e9: Status 404 returned error can't find the container with id f947ee0ff506133f29a8d193fa40d428802dff1a65bc071679ff0ae64b6694e9 Oct 30 13:06:03.545017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.544983 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" Oct 30 13:06:03.545456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.545023 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2" event={"ID":"2acfffb1-ff93-4bb6-a30c-d27bbc1328b8","Type":"ContainerDied","Data":"8efb25ac47bb8ed432d8842bf2f0fb79082f9180d13790d21eeb777d9980a370"} Oct 30 13:06:03.545456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.545110 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8efb25ac47bb8ed432d8842bf2f0fb79082f9180d13790d21eeb777d9980a370" Oct 30 13:06:03.547256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.547222 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" event={"ID":"46c410dc-efe9-4661-aa8d-bc27ddd62c16","Type":"ContainerStarted","Data":"487f155047b72d3b5c03ed52c245ca995a1337b4e63ef84a882ae813a61ad2d2"} Oct 30 13:06:03.547394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.547273 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" event={"ID":"46c410dc-efe9-4661-aa8d-bc27ddd62c16","Type":"ContainerStarted","Data":"f947ee0ff506133f29a8d193fa40d428802dff1a65bc071679ff0ae64b6694e9"} Oct 30 13:06:03.583924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.583873 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" podStartSLOduration=1.5838601190000001 podStartE2EDuration="1.583860119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:03.583579092 +0000 UTC m=+870.398773496" watchObservedRunningTime="2026-10-30 13:06:03.583860119 +0000 UTC m=+870.399054579" Oct 30 13:06:03.617723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.617687 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:03.909183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.909145 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s"] Oct 30 13:06:03.909900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.909875 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2acfffb1-ff93-4bb6-a30c-d27bbc1328b8" containerName="revoke-keys" Oct 30 13:06:03.909900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.909900 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acfffb1-ff93-4bb6-a30c-d27bbc1328b8" containerName="revoke-keys" Oct 30 13:06:03.910093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.910040 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acfffb1-ff93-4bb6-a30c-d27bbc1328b8" containerName="revoke-keys" Oct 30 13:06:03.913575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.913552 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:03.941901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.941865 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s"] Oct 30 13:06:03.949569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.949365 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:03.949569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.949409 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:03.949569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.949465 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:03.949569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.949492 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52lvj\" (UniqueName: \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-kube-api-access-52lvj\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:03.949569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.949519 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:03.950396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.950361 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:03.951027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.950452 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:03.951027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.950501 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:03.951027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.950541 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:03.956837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.951457 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq"] Oct 30 13:06:03.960089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.960069 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:03.979815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:03.979790 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq"] Oct 30 13:06:04.051402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051361 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.051402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051401 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.051627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051494 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.051627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051579 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52lvj\" (UniqueName: \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-kube-api-access-52lvj\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.051627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051615 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.051772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051672 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.051772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051699 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.051772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051724 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.051916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051804 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.051975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051904 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.052030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.051970 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.052030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052022 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963fea52-5fa2-4b2e-be70-c18f8621f46c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.052141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052053 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.052141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052091 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.052141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052119 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.052290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052215 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.052290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052282 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqsxb\" (UniqueName: \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-kube-api-access-dqsxb\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.052397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052372 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.052397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052372 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.052485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052415 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.052657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052636 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.052743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052675 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.052743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.052729 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.054176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.054152 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.054888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.054869 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.072788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.072761 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.073007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.072985 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52lvj\" (UniqueName: \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-kube-api-access-52lvj\") pod \"maas-default-gateway-openshift-default-64d4cc998c-74k2s\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.153185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153144 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153197 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqsxb\" (UniqueName: \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-kube-api-access-dqsxb\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153237 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153254 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153295 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153359 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153609 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153717 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153797 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153842 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153896 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963fea52-5fa2-4b2e-be70-c18f8621f46c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.153970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.153922 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.154211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.154142 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.154552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.154535 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963fea52-5fa2-4b2e-be70-c18f8621f46c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.156039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.156016 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.156385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.156358 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.174710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.174658 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqsxb\" (UniqueName: \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-kube-api-access-dqsxb\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.179879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.179856 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-lzglq\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.229073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.229035 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:04.274308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.274273 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:04.554739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.554714 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s"] Oct 30 13:06:04.557803 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:06:04.557768 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a1172f_16fa_43a9_8c23_bfcd541fbc0f.slice/crio-62522da1222d908b34cf6617a354a1ea1ef7d513bcffaa72a744dbe8b151f336 WatchSource:0}: Error finding container 62522da1222d908b34cf6617a354a1ea1ef7d513bcffaa72a744dbe8b151f336: Status 404 returned error can't find the container with id 62522da1222d908b34cf6617a354a1ea1ef7d513bcffaa72a744dbe8b151f336 Oct 30 13:06:04.558853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.558829 5049 generic.go:356] "Generic (PLEG): container finished" podID="46c410dc-efe9-4661-aa8d-bc27ddd62c16" containerID="487f155047b72d3b5c03ed52c245ca995a1337b4e63ef84a882ae813a61ad2d2" exitCode=0 Oct 30 13:06:04.558937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.558905 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" event={"ID":"46c410dc-efe9-4661-aa8d-bc27ddd62c16","Type":"ContainerDied","Data":"487f155047b72d3b5c03ed52c245ca995a1337b4e63ef84a882ae813a61ad2d2"} Oct 30 13:06:04.560054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.560027 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:04.560127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.560087 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:04.560127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.560118 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:04.605892 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:06:04.605853 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963fea52_5fa2_4b2e_be70_c18f8621f46c.slice/crio-92085ff4aa4dc407d58284d3570ae93bf5396dfe762804fca3867936f239d83f WatchSource:0}: Error finding container 92085ff4aa4dc407d58284d3570ae93bf5396dfe762804fca3867936f239d83f: Status 404 returned error can't find the container with id 92085ff4aa4dc407d58284d3570ae93bf5396dfe762804fca3867936f239d83f Oct 30 13:06:04.609385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.609349 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:04.609493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.609439 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:04.609493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.609486 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:04.625372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:04.620879 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq"] Oct 30 13:06:05.504316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:05.504281 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:05.626910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:05.626860 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" event={"ID":"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f","Type":"ContainerStarted","Data":"bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92"} Oct 30 13:06:05.627549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:05.627511 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" event={"ID":"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f","Type":"ContainerStarted","Data":"62522da1222d908b34cf6617a354a1ea1ef7d513bcffaa72a744dbe8b151f336"} Oct 30 13:06:05.630554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:05.630505 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" event={"ID":"963fea52-5fa2-4b2e-be70-c18f8621f46c","Type":"ContainerStarted","Data":"b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476"} Oct 30 13:06:05.630687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:05.630578 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" event={"ID":"963fea52-5fa2-4b2e-be70-c18f8621f46c","Type":"ContainerStarted","Data":"92085ff4aa4dc407d58284d3570ae93bf5396dfe762804fca3867936f239d83f"} Oct 30 13:06:05.755429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:05.749140 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" podStartSLOduration=2.74911976 podStartE2EDuration="2.74911976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:05.726957284 +0000 UTC m=+872.542151708" watchObservedRunningTime="2026-10-30 13:06:05.74911976 +0000 UTC m=+872.564314183" Oct 30 13:06:05.755429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:05.749676 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" podStartSLOduration=2.7496668890000002 podStartE2EDuration="2.749666889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:05.679763303 +0000 UTC m=+872.494957739" watchObservedRunningTime="2026-10-30 13:06:05.749666889 +0000 UTC m=+872.564861313" Oct 30 13:06:06.397396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.244515 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:06.513359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.468863 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:06.513359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.431714 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-74k2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:06:06.513359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.469718 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:06:06.577852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.570255 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-lzglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:06:06.577852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.570384 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:06:06.709346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.708407 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" event={"ID":"46c410dc-efe9-4661-aa8d-bc27ddd62c16","Type":"ContainerDied","Data":"f947ee0ff506133f29a8d193fa40d428802dff1a65bc071679ff0ae64b6694e9"} Oct 30 13:06:06.709346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.708580 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f947ee0ff506133f29a8d193fa40d428802dff1a65bc071679ff0ae64b6694e9" Oct 30 13:06:06.737519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.737460 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:06.756955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.756924 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:06.799345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.798481 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:06.799563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.799390 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:06.862354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.860793 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-openshift-ca\") pod \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " Oct 30 13:06:06.862354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.860893 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-xks-ca\") pod \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " Oct 30 13:06:06.862354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.860931 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c410dc-efe9-4661-aa8d-bc27ddd62c16-kube-api-access-gfl98\" (UniqueName: \"kubernetes.io/projected/46c410dc-efe9-4661-aa8d-bc27ddd62c16-kube-api-access-gfl98\") pod \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\" (UID: \"46c410dc-efe9-4661-aa8d-bc27ddd62c16\") " Oct 30 13:06:06.871596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.871546 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-openshift-ca" pod "46c410dc-efe9-4661-aa8d-bc27ddd62c16" (UID: "46c410dc-efe9-4661-aa8d-bc27ddd62c16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:07.021406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:06.966198 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.021406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.017429 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:06:07.121356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.118173 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:07.143351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.132787 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c410dc-efe9-4661-aa8d-bc27ddd62c16-kube-api-access-gfl98" pod "46c410dc-efe9-4661-aa8d-bc27ddd62c16" (UID: "46c410dc-efe9-4661-aa8d-bc27ddd62c16"). InnerVolumeSpecName "kube-api-access-gfl98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:07.143351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.132876 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-xks-ca" pod "46c410dc-efe9-4661-aa8d-bc27ddd62c16" (UID: "46c410dc-efe9-4661-aa8d-bc27ddd62c16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:07.273406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.242222 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-74k2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:06:07.273712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.273672 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:06:07.273875 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.243341 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:07.274076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.244166 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c410dc-efe9-4661-aa8d-bc27ddd62c16-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.274162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.274100 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfl98\" (UniqueName: \"kubernetes.io/projected/46c410dc-efe9-4661-aa8d-bc27ddd62c16-kube-api-access-gfl98\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:07.274162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.243277 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:07.291369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.275134 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-lzglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:06:07.291369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.275173 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:06:07.689342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.648723 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:07.726902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.726864 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:07.742559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.742517 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:06:07.781747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.776231 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z" Oct 30 13:06:07.825557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.791425 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:06:07.825557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.805832 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:06:07.825557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:07.821431 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:06:08.023355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.022722 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:06:08.023604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.023580 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:06:08.252430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.234486 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-74k2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:06:08.252430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.234548 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:06:08.297155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.297095 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:06:08.301921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.301886 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:06:08.311354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.308422 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-lzglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:06:08.311354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.308474 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:06:08.358902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.358848 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:08.381214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.381182 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:06:08.499347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.493067 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:06:08.562599 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:06:08.562558 5049 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c410dc_efe9_4661_aa8d_bc27ddd62c16.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:06:08.656958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.654600 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:08.656958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.655929 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:06:08.804344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.783235 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:08.836473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.822752 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:06:08.993369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:08.974440 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Oct 30 13:06:09.029198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.028574 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:06:09.252197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.252106 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-74k2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:06:09.252479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.252437 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:06:09.288876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.288765 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-lzglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:06:09.288876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.288823 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:06:09.302173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.302133 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:09.329232 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.329164 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:06:09.491238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.490755 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:06:09.491238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.490994 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:06:09.491238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.491111 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:06:09.501564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.501184 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:06:09.983933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:09.983895 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.234406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:10.233467 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-74k2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:06:10.234406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:10.233528 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:06:10.275100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:10.275021 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-lzglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:06:10.275100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:10.275099 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:06:11.233346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:11.232701 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-74k2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:06:11.233346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:11.232766 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:06:11.274831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:11.274792 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-lzglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:06:11.275005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:11.274846 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:06:12.233345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.233083 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-74k2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:06:12.233345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.233145 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:06:12.275536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.275498 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-lzglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:06:12.275694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.275571 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:06:12.672855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.672814 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9"] Oct 30 13:06:12.673750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.673726 5049 cpu_manager.go:414] "containerMap: removing container" podUID="46c410dc-efe9-4661-aa8d-bc27ddd62c16" containerName="revoke-keys" Oct 30 13:06:12.673750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.673750 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c410dc-efe9-4661-aa8d-bc27ddd62c16" containerName="revoke-keys" Oct 30 13:06:12.673929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.673888 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c410dc-efe9-4661-aa8d-bc27ddd62c16" containerName="revoke-keys" Oct 30 13:06:12.678174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.678154 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:12.700217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.700185 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9"] Oct 30 13:06:12.704843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.704820 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:06:12.705030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.704830 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:12.725822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.725775 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63da8efd-drpx9\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:12.725980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.725965 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63da8efd-drpx9\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:12.726091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.726073 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn4rl\" (UniqueName: \"kubernetes.io/projected/d165b957-19eb-4710-998c-7bfa4cf683b8-kube-api-access-sn4rl\") pod \"maas-api-revoke-sub-63da8efd-drpx9\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:12.795295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.795210 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:12.827956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.827913 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63da8efd-drpx9\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:12.828173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.828071 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63da8efd-drpx9\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:12.828173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.828155 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn4rl\" (UniqueName: \"kubernetes.io/projected/d165b957-19eb-4710-998c-7bfa4cf683b8-kube-api-access-sn4rl\") pod \"maas-api-revoke-sub-63da8efd-drpx9\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:12.828876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.828850 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63da8efd-drpx9\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:12.831687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.831645 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63da8efd-drpx9\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:12.850290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.850257 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn4rl\" (UniqueName: \"kubernetes.io/projected/d165b957-19eb-4710-998c-7bfa4cf683b8-kube-api-access-sn4rl\") pod \"maas-api-revoke-sub-63da8efd-drpx9\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:12.993448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:12.993365 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:13.229655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:13.229616 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-74k2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:06:13.229830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:13.229679 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 13:06:13.275791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:13.275711 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-lzglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:06:13.275791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:13.275755 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 13:06:13.323976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:13.323937 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9"] Oct 30 13:06:13.325185 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:06:13.325110 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd165b957_19eb_4710_998c_7bfa4cf683b8.slice/crio-f06e107517d47eaeaeeff859eeaaf6a9e3b35d501aa2d63bd3ddffe8dffe56dc WatchSource:0}: Error finding container f06e107517d47eaeaeeff859eeaaf6a9e3b35d501aa2d63bd3ddffe8dffe56dc: Status 404 returned error can't find the container with id f06e107517d47eaeaeeff859eeaaf6a9e3b35d501aa2d63bd3ddffe8dffe56dc Oct 30 13:06:13.905279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:13.905244 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" event={"ID":"d165b957-19eb-4710-998c-7bfa4cf683b8","Type":"ContainerStarted","Data":"62057525f7456970587703923ac57f41084073a6412af20e11cc63bf5b8a716d"} Oct 30 13:06:13.905472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:13.905308 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" event={"ID":"d165b957-19eb-4710-998c-7bfa4cf683b8","Type":"ContainerStarted","Data":"f06e107517d47eaeaeeff859eeaaf6a9e3b35d501aa2d63bd3ddffe8dffe56dc"} Oct 30 13:06:13.946738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:13.946683 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" podStartSLOduration=1.9466700000000001 podStartE2EDuration="1.94667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:13.942432391 +0000 UTC m=+880.757626815" watchObservedRunningTime="2026-10-30 13:06:13.94667 +0000 UTC m=+880.761864422" Oct 30 13:06:14.841845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:14.841810 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:14.917137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:14.917097 5049 generic.go:356] "Generic (PLEG): container finished" podID="d165b957-19eb-4710-998c-7bfa4cf683b8" containerID="62057525f7456970587703923ac57f41084073a6412af20e11cc63bf5b8a716d" exitCode=0 Oct 30 13:06:14.917377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:14.917343 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" event={"ID":"d165b957-19eb-4710-998c-7bfa4cf683b8","Type":"ContainerDied","Data":"62057525f7456970587703923ac57f41084073a6412af20e11cc63bf5b8a716d"} Oct 30 13:06:15.024359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:15.024314 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:15.025593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:15.025569 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:15.026306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:15.026283 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:06:15.059929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:15.059898 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:15.060366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:15.060314 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:15.061398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:15.061373 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:06:16.079572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.079540 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:16.181340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.181249 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d165b957-19eb-4710-998c-7bfa4cf683b8-kube-api-access-sn4rl\" (UniqueName: \"kubernetes.io/projected/d165b957-19eb-4710-998c-7bfa4cf683b8-kube-api-access-sn4rl\") pod \"d165b957-19eb-4710-998c-7bfa4cf683b8\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " Oct 30 13:06:16.181340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.181305 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-xks-ca\") pod \"d165b957-19eb-4710-998c-7bfa4cf683b8\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " Oct 30 13:06:16.181537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.181372 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-openshift-ca\") pod \"d165b957-19eb-4710-998c-7bfa4cf683b8\" (UID: \"d165b957-19eb-4710-998c-7bfa4cf683b8\") " Oct 30 13:06:16.181801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.181769 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-openshift-ca" pod "d165b957-19eb-4710-998c-7bfa4cf683b8" (UID: "d165b957-19eb-4710-998c-7bfa4cf683b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:16.183393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.183361 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d165b957-19eb-4710-998c-7bfa4cf683b8-kube-api-access-sn4rl" pod "d165b957-19eb-4710-998c-7bfa4cf683b8" (UID: "d165b957-19eb-4710-998c-7bfa4cf683b8"). InnerVolumeSpecName "kube-api-access-sn4rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:16.183489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.183440 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-xks-ca" pod "d165b957-19eb-4710-998c-7bfa4cf683b8" (UID: "d165b957-19eb-4710-998c-7bfa4cf683b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:16.282453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.282423 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn4rl\" (UniqueName: \"kubernetes.io/projected/d165b957-19eb-4710-998c-7bfa4cf683b8-kube-api-access-sn4rl\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:16.282453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.282450 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:16.282453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.282461 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d165b957-19eb-4710-998c-7bfa4cf683b8-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:06:16.819202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.819172 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.940073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.940040 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" Oct 30 13:06:16.940225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.940082 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9" event={"ID":"d165b957-19eb-4710-998c-7bfa4cf683b8","Type":"ContainerDied","Data":"f06e107517d47eaeaeeff859eeaaf6a9e3b35d501aa2d63bd3ddffe8dffe56dc"} Oct 30 13:06:16.940225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:16.940150 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f06e107517d47eaeaeeff859eeaaf6a9e3b35d501aa2d63bd3ddffe8dffe56dc" Oct 30 13:06:18.611869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:18.611823 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:20.450277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:20.450239 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:22.416589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:22.416541 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:22.904870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:22.904836 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:24.728336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:24.728298 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:24.888814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:24.888779 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:26.676429 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:06:26.676388 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:06:26.739973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:26.739893 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:28.550120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:28.550080 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:30.640122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:30.640087 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:32.453513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:32.453474 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:33.433279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:33.433248 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:06:33.433957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:33.433934 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:06:34.201880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:34.201839 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:36.230118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:36.230066 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:38.006791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.006759 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:38.874856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.874823 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s"] Oct 30 13:06:38.875434 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.875419 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d165b957-19eb-4710-998c-7bfa4cf683b8" containerName="revoke-keys" Oct 30 13:06:38.875487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.875437 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d165b957-19eb-4710-998c-7bfa4cf683b8" containerName="revoke-keys" Oct 30 13:06:38.875570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.875559 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d165b957-19eb-4710-998c-7bfa4cf683b8" containerName="revoke-keys" Oct 30 13:06:38.880721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.880698 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:38.903776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.903745 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"kube-root-ca.crt\"" Oct 30 13:06:38.903918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.903791 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"openshift-service-ca.crt\"" Oct 30 13:06:38.903985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.903930 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"distinct-w5-e34fd5-kserve-self-signed-certs\"" Oct 30 13:06:38.904115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.904100 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"default-dockercfg-vgqrm\"" Oct 30 13:06:38.907466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.907440 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s"] Oct 30 13:06:38.962591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.962561 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcmgr\" (UniqueName: \"kubernetes.io/projected/d191137c-94a6-4f18-b4f9-1c2e85de46de-kube-api-access-mcmgr\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:38.962739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.962602 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-model-cache\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:38.962739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.962726 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d191137c-94a6-4f18-b4f9-1c2e85de46de-tls-certs\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:38.962839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.962822 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-home\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:38.962880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.962855 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-tmp-dir\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:38.962921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.962880 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-dshm\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:38.983737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:38.983705 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:39.063896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.063865 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d191137c-94a6-4f18-b4f9-1c2e85de46de-tls-certs\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.064283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.063925 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-home\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.064283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.063945 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-tmp-dir\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.064283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.063962 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-dshm\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.064283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.064052 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmgr\" (UniqueName: \"kubernetes.io/projected/d191137c-94a6-4f18-b4f9-1c2e85de46de-kube-api-access-mcmgr\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.064283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.064080 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-model-cache\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.064618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.064524 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-tmp-dir\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.064618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.064543 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-home\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.064618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.064603 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-model-cache\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.066703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.066683 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-dshm\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.066937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.066921 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d191137c-94a6-4f18-b4f9-1c2e85de46de-tls-certs\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.089255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.089194 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmgr\" (UniqueName: \"kubernetes.io/projected/d191137c-94a6-4f18-b4f9-1c2e85de46de-kube-api-access-mcmgr\") pod \"distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.205289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.205208 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:39.510657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.510633 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s"] Oct 30 13:06:39.513360 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:06:39.513308 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd191137c_94a6_4f18_b4f9_1c2e85de46de.slice/crio-9d73640b6d5a89e9feda1c3b9971120342d411ef57c614de9669c8abf374ef55 WatchSource:0}: Error finding container 9d73640b6d5a89e9feda1c3b9971120342d411ef57c614de9669c8abf374ef55: Status 404 returned error can't find the container with id 9d73640b6d5a89e9feda1c3b9971120342d411ef57c614de9669c8abf374ef55 Oct 30 13:06:39.946524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:39.946486 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:40.137954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:40.137920 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" event={"ID":"d191137c-94a6-4f18-b4f9-1c2e85de46de","Type":"ContainerStarted","Data":"7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5"} Oct 30 13:06:40.138456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:40.137962 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" event={"ID":"d191137c-94a6-4f18-b4f9-1c2e85de46de","Type":"ContainerStarted","Data":"9d73640b6d5a89e9feda1c3b9971120342d411ef57c614de9669c8abf374ef55"} Oct 30 13:06:40.190515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:40.190464 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podStartSLOduration=2.190450193 podStartE2EDuration="2.190450193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:40.187537106 +0000 UTC m=+907.002731529" watchObservedRunningTime="2026-10-30 13:06:40.190450193 +0000 UTC m=+907.005644615" Oct 30 13:06:40.206419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:40.206344 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:41.786438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:41.786397 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:43.795473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:43.795436 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:45.935256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:45.935221 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:48.060159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:48.060115 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:49.205489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:49.205459 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:49.219412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:49.219386 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:49.806194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:49.806156 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:50.219480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:50.219390 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:06:51.687346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:51.687262 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:52.016532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:52.016447 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:53.749277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:53.749241 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:55.561070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:55.561000 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:56.676938 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:06:56.676900 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:06:57.372740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:57.372699 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:59.390191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:06:59.390160 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:01.322726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:01.322692 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:03.318952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.318909 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:03.737723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.737650 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v"] Oct 30 13:07:03.742500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.742483 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.764645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.764619 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v"] Oct 30 13:07:03.773411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.773386 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"distinct-2-w4-d7e62b-kserve-self-signed-certs\"" Oct 30 13:07:03.777981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.777958 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-dshm\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.778123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.777985 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b965cb-83d4-4c13-a4be-77060bfca0cc-tls-certs\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.778123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.778020 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-home\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.778123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.778037 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt64m\" (UniqueName: \"kubernetes.io/projected/76b965cb-83d4-4c13-a4be-77060bfca0cc-kube-api-access-wt64m\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.778358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.778273 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-model-cache\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.778440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.778375 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-tmp-dir\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.851911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.851879 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:03.879193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.879163 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-dshm\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.879193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.879196 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b965cb-83d4-4c13-a4be-77060bfca0cc-tls-certs\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.879398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.879231 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-home\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.879398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.879345 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt64m\" (UniqueName: \"kubernetes.io/projected/76b965cb-83d4-4c13-a4be-77060bfca0cc-kube-api-access-wt64m\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.879495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.879474 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-model-cache\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.879540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.879530 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-tmp-dir\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.879660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.879634 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-home\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.879828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.879806 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-model-cache\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.879882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.879867 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-tmp-dir\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.882121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.882097 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b965cb-83d4-4c13-a4be-77060bfca0cc-tls-certs\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.882703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.882683 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-dshm\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:03.902939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:03.902911 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt64m\" (UniqueName: \"kubernetes.io/projected/76b965cb-83d4-4c13-a4be-77060bfca0cc-kube-api-access-wt64m\") pod \"distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:04.061401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:04.061367 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:04.371410 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:07:04.371380 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76b965cb_83d4_4c13_a4be_77060bfca0cc.slice/crio-a96ee9a530b5eca1f143d379894adcd73186265fe0bcc6c862fd0dba4a3cfeb6 WatchSource:0}: Error finding container a96ee9a530b5eca1f143d379894adcd73186265fe0bcc6c862fd0dba4a3cfeb6: Status 404 returned error can't find the container with id a96ee9a530b5eca1f143d379894adcd73186265fe0bcc6c862fd0dba4a3cfeb6 Oct 30 13:07:04.372268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:04.372240 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v"] Oct 30 13:07:05.074902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:05.074862 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:05.226172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:05.226135 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:05.341375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:05.341274 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" event={"ID":"76b965cb-83d4-4c13-a4be-77060bfca0cc","Type":"ContainerStarted","Data":"ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9"} Oct 30 13:07:05.341375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:05.341314 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" event={"ID":"76b965cb-83d4-4c13-a4be-77060bfca0cc","Type":"ContainerStarted","Data":"a96ee9a530b5eca1f143d379894adcd73186265fe0bcc6c862fd0dba4a3cfeb6"} Oct 30 13:07:05.395651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:05.395593 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podStartSLOduration=2.395577207 podStartE2EDuration="2.395577207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:05.383399091 +0000 UTC m=+932.198593516" watchObservedRunningTime="2026-10-30 13:07:05.395577207 +0000 UTC m=+932.210771629" Oct 30 13:07:06.062051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:06.062017 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:07.005984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.005951 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:07.844252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.844208 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb"] Oct 30 13:07:07.849546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.849522 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:07.876927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.876898 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb"] Oct 30 13:07:07.932585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.932552 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6843e1-fc62-4312-8130-019a30b2a4af-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:07.932751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.932592 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkd2n\" (UniqueName: \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-kube-api-access-xkd2n\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:07.932751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.932728 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:07.932829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.932788 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:07.932885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.932867 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:07.932979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.932967 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:07.933016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.932992 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:07.933050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.933026 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:07.933084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:07.933061 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.034113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.034076 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.034623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.034145 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.034623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.034180 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.034623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.034226 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.034623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.034248 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.034623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.034378 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6843e1-fc62-4312-8130-019a30b2a4af-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.034623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.034411 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkd2n\" (UniqueName: \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-kube-api-access-xkd2n\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.034623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.034502 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.034623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.034532 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.035048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.034660 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.035109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.035055 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.035109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.035088 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.035523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.035493 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6843e1-fc62-4312-8130-019a30b2a4af-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.035659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.035598 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.037275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.037250 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.037552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.037534 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.059610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.059579 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.059788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.059766 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkd2n\" (UniqueName: \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-kube-api-access-xkd2n\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.165159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.165085 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:08.554352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.554312 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb"] Oct 30 13:07:08.554586 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:07:08.554563 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6843e1_fc62_4312_8130_019a30b2a4af.slice/crio-322624ec35f573383f97cd68690dbc8a52a711ad1e52387ca37268d1bf710d61 WatchSource:0}: Error finding container 322624ec35f573383f97cd68690dbc8a52a711ad1e52387ca37268d1bf710d61: Status 404 returned error can't find the container with id 322624ec35f573383f97cd68690dbc8a52a711ad1e52387ca37268d1bf710d61 Oct 30 13:07:08.557523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.557495 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:08.557600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.557557 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:08.557645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:08.557619 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:09.016166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:09.016078 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:09.370198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:09.370160 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" event={"ID":"0b6843e1-fc62-4312-8130-019a30b2a4af","Type":"ContainerStarted","Data":"f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed"} Oct 30 13:07:09.370594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:09.370215 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" event={"ID":"0b6843e1-fc62-4312-8130-019a30b2a4af","Type":"ContainerStarted","Data":"322624ec35f573383f97cd68690dbc8a52a711ad1e52387ca37268d1bf710d61"} Oct 30 13:07:09.438494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:09.438432 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" podStartSLOduration=2.438416889 podStartE2EDuration="2.438416889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:09.435640897 +0000 UTC m=+936.250835321" watchObservedRunningTime="2026-10-30 13:07:09.438416889 +0000 UTC m=+936.253611313" Oct 30 13:07:10.165744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:10.165701 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:10.167145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:10.167118 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:07:10.167269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:10.167166 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 13:07:10.937346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:10.937059 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:11.165540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:11.165507 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:07:11.165715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:11.165567 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 13:07:12.165736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:12.165700 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:07:12.166142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:12.165762 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 13:07:13.130345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:13.126899 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:13.166076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:13.166045 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:07:13.166537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:13.166115 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 13:07:14.061771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.061730 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:14.081743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.081694 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:14.165935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.165836 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:07:14.165935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.165900 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 13:07:14.445277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.445188 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:07:14.502740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.502706 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6"] Oct 30 13:07:14.526743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.515593 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:14.548715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.548689 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:14.551371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.549894 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:07:14.555287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.555256 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6"] Oct 30 13:07:14.645085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.645034 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485de4c0-2624-4e32-b606-18088c985410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1ab4ee-dn9m6\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:14.645334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.645299 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xfr\" (UniqueName: \"kubernetes.io/projected/485de4c0-2624-4e32-b606-18088c985410-kube-api-access-79xfr\") pod \"maas-api-revoke-sub-ff1ab4ee-dn9m6\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:14.645423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.645404 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485de4c0-2624-4e32-b606-18088c985410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1ab4ee-dn9m6\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:14.746376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.746263 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xfr\" (UniqueName: \"kubernetes.io/projected/485de4c0-2624-4e32-b606-18088c985410-kube-api-access-79xfr\") pod \"maas-api-revoke-sub-ff1ab4ee-dn9m6\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:14.746376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.746351 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485de4c0-2624-4e32-b606-18088c985410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1ab4ee-dn9m6\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:14.746601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.746427 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485de4c0-2624-4e32-b606-18088c985410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1ab4ee-dn9m6\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:14.747093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.747054 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485de4c0-2624-4e32-b606-18088c985410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1ab4ee-dn9m6\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:14.749423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.749400 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485de4c0-2624-4e32-b606-18088c985410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1ab4ee-dn9m6\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:14.769884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.769844 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xfr\" (UniqueName: \"kubernetes.io/projected/485de4c0-2624-4e32-b606-18088c985410-kube-api-access-79xfr\") pod \"maas-api-revoke-sub-ff1ab4ee-dn9m6\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:14.857282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:14.857252 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:15.116023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:15.115985 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:15.165627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:15.165592 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:07:15.165824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:15.165639 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 13:07:15.183939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:15.183912 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6"] Oct 30 13:07:15.186540 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:07:15.186508 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485de4c0_2624_4e32_b606_18088c985410.slice/crio-5461800a5ef061711f70b658d5c99d362129f32b15733bb9855ea16b72702813 WatchSource:0}: Error finding container 5461800a5ef061711f70b658d5c99d362129f32b15733bb9855ea16b72702813: Status 404 returned error can't find the container with id 5461800a5ef061711f70b658d5c99d362129f32b15733bb9855ea16b72702813 Oct 30 13:07:15.416823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:15.416738 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" event={"ID":"485de4c0-2624-4e32-b606-18088c985410","Type":"ContainerStarted","Data":"5461800a5ef061711f70b658d5c99d362129f32b15733bb9855ea16b72702813"} Oct 30 13:07:16.430156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:16.430116 5049 generic.go:356] "Generic (PLEG): container finished" podID="485de4c0-2624-4e32-b606-18088c985410" containerID="646019a2a0e075f9d7a732e9978408f5eb795733fec217be29996e64d0f42395" exitCode=0 Oct 30 13:07:16.430737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:16.430215 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" event={"ID":"485de4c0-2624-4e32-b606-18088c985410","Type":"ContainerDied","Data":"646019a2a0e075f9d7a732e9978408f5eb795733fec217be29996e64d0f42395"} Oct 30 13:07:17.025999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.025965 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:17.026429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.026407 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:17.028145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.028125 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:07:17.066616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.066573 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:17.598076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.598053 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:17.686833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.686801 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/485de4c0-2624-4e32-b606-18088c985410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485de4c0-2624-4e32-b606-18088c985410-maas-api-xks-ca\") pod \"485de4c0-2624-4e32-b606-18088c985410\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " Oct 30 13:07:17.686970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.686890 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485de4c0-2624-4e32-b606-18088c985410-kube-api-access-79xfr\" (UniqueName: \"kubernetes.io/projected/485de4c0-2624-4e32-b606-18088c985410-kube-api-access-79xfr\") pod \"485de4c0-2624-4e32-b606-18088c985410\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " Oct 30 13:07:17.687039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.687001 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/485de4c0-2624-4e32-b606-18088c985410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485de4c0-2624-4e32-b606-18088c985410-maas-api-openshift-ca\") pod \"485de4c0-2624-4e32-b606-18088c985410\" (UID: \"485de4c0-2624-4e32-b606-18088c985410\") " Oct 30 13:07:17.687469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.687402 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485de4c0-2624-4e32-b606-18088c985410-maas-api-openshift-ca" pod "485de4c0-2624-4e32-b606-18088c985410" (UID: "485de4c0-2624-4e32-b606-18088c985410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:17.688986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.688963 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/485de4c0-2624-4e32-b606-18088c985410-maas-api-xks-ca" pod "485de4c0-2624-4e32-b606-18088c985410" (UID: "485de4c0-2624-4e32-b606-18088c985410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:17.689081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.689033 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485de4c0-2624-4e32-b606-18088c985410-kube-api-access-79xfr" pod "485de4c0-2624-4e32-b606-18088c985410" (UID: "485de4c0-2624-4e32-b606-18088c985410"). InnerVolumeSpecName "kube-api-access-79xfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:17.788858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.788825 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485de4c0-2624-4e32-b606-18088c985410-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:17.788858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.788853 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485de4c0-2624-4e32-b606-18088c985410-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:17.788858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:17.788863 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79xfr\" (UniqueName: \"kubernetes.io/projected/485de4c0-2624-4e32-b606-18088c985410-kube-api-access-79xfr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:18.444259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:18.444226 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" Oct 30 13:07:18.444447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:18.444261 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6" event={"ID":"485de4c0-2624-4e32-b606-18088c985410","Type":"ContainerDied","Data":"5461800a5ef061711f70b658d5c99d362129f32b15733bb9855ea16b72702813"} Oct 30 13:07:18.444447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:18.444336 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5461800a5ef061711f70b658d5c99d362129f32b15733bb9855ea16b72702813" Oct 30 13:07:18.808892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:18.808848 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:19.099447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:19.099345 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:20.837108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:20.837066 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:22.620752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:22.620673 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:24.431151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:24.431069 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:24.939114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:24.939081 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:26.080025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.079981 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr"] Oct 30 13:07:26.080643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.080625 5049 cpu_manager.go:414] "containerMap: removing container" podUID="485de4c0-2624-4e32-b606-18088c985410" containerName="revoke-keys" Oct 30 13:07:26.080692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.080644 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485de4c0-2624-4e32-b606-18088c985410" containerName="revoke-keys" Oct 30 13:07:26.080768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.080758 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="485de4c0-2624-4e32-b606-18088c985410" containerName="revoke-keys" Oct 30 13:07:26.083664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.083646 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.087538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.087512 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a050ff-29fqr\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.087673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.087574 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a050ff-29fqr\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.087720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.087669 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lww6t\" (UniqueName: \"kubernetes.io/projected/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-kube-api-access-lww6t\") pod \"maas-api-revoke-sub-09a050ff-29fqr\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.100227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.100206 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:26.100580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.100560 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:07:26.118505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.118470 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr"] Oct 30 13:07:26.189017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.188981 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a050ff-29fqr\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.189182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.189085 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a050ff-29fqr\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.189182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.189153 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lww6t\" (UniqueName: \"kubernetes.io/projected/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-kube-api-access-lww6t\") pod \"maas-api-revoke-sub-09a050ff-29fqr\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.189797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.189773 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a050ff-29fqr\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.191773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.191747 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a050ff-29fqr\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.213178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.213146 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lww6t\" (UniqueName: \"kubernetes.io/projected/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-kube-api-access-lww6t\") pod \"maas-api-revoke-sub-09a050ff-29fqr\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.298026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.297990 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:26.397362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.397250 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:26.677084 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:07:26.677008 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:07:26.720334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:26.720294 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr"] Oct 30 13:07:26.723155 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:07:26.723112 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd1dc37_2b68_44e9_8b82_d836ab18a7a0.slice/crio-93723800fd12fcd7f42d512feb0beecfeead524863e240ac9d5814293db8e66e WatchSource:0}: Error finding container 93723800fd12fcd7f42d512feb0beecfeead524863e240ac9d5814293db8e66e: Status 404 returned error can't find the container with id 93723800fd12fcd7f42d512feb0beecfeead524863e240ac9d5814293db8e66e Oct 30 13:07:27.512078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:27.512037 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" event={"ID":"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0","Type":"ContainerStarted","Data":"66711922c01b23a4eb0fc60e6b80d1539c46a9fc6793783f80d450245bc9addc"} Oct 30 13:07:27.512558 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:27.512116 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" event={"ID":"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0","Type":"ContainerStarted","Data":"93723800fd12fcd7f42d512feb0beecfeead524863e240ac9d5814293db8e66e"} Oct 30 13:07:27.556171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:27.556116 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" podStartSLOduration=1.556097901 podStartE2EDuration="1.556097901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:27.552873178 +0000 UTC m=+954.368067602" watchObservedRunningTime="2026-10-30 13:07:27.556097901 +0000 UTC m=+954.371292349" Oct 30 13:07:28.186997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:28.186966 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:28.521106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:28.521019 5049 generic.go:356] "Generic (PLEG): container finished" podID="5bd1dc37-2b68-44e9-8b82-d836ab18a7a0" containerID="66711922c01b23a4eb0fc60e6b80d1539c46a9fc6793783f80d450245bc9addc" exitCode=0 Oct 30 13:07:28.521509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:28.521098 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" event={"ID":"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0","Type":"ContainerDied","Data":"66711922c01b23a4eb0fc60e6b80d1539c46a9fc6793783f80d450245bc9addc"} Oct 30 13:07:29.745476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:29.745445 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:29.839633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:29.839593 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-openshift-ca\") pod \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " Oct 30 13:07:29.839957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:29.839931 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-xks-ca\") pod \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " Oct 30 13:07:29.840073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:29.839988 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-kube-api-access-lww6t\" (UniqueName: \"kubernetes.io/projected/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-kube-api-access-lww6t\") pod \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\" (UID: \"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0\") " Oct 30 13:07:29.842722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:29.842676 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-kube-api-access-lww6t" pod "5bd1dc37-2b68-44e9-8b82-d836ab18a7a0" (UID: "5bd1dc37-2b68-44e9-8b82-d836ab18a7a0"). InnerVolumeSpecName "kube-api-access-lww6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:29.842722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:29.842695 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-openshift-ca" pod "5bd1dc37-2b68-44e9-8b82-d836ab18a7a0" (UID: "5bd1dc37-2b68-44e9-8b82-d836ab18a7a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:29.844563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:29.844534 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-xks-ca" pod "5bd1dc37-2b68-44e9-8b82-d836ab18a7a0" (UID: "5bd1dc37-2b68-44e9-8b82-d836ab18a7a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:29.945409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:29.945289 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:29.945409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:29.945349 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lww6t\" (UniqueName: \"kubernetes.io/projected/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-kube-api-access-lww6t\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:29.945409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:29.945366 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:30.244065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:30.243967 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:30.541580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:30.541492 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" Oct 30 13:07:30.541580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:30.541495 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr" event={"ID":"5bd1dc37-2b68-44e9-8b82-d836ab18a7a0","Type":"ContainerDied","Data":"93723800fd12fcd7f42d512feb0beecfeead524863e240ac9d5814293db8e66e"} Oct 30 13:07:30.541765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:30.541611 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93723800fd12fcd7f42d512feb0beecfeead524863e240ac9d5814293db8e66e" Oct 30 13:07:32.094664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:32.094620 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:33.851094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:33.851049 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:35.405883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:35.405844 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:35.832681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:35.832629 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:37.630183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:37.630144 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:38.924241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:38.924174 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:39.087766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:39.087720 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:39.430014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:39.429977 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:40.875602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.875565 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26"] Oct 30 13:07:40.876132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.876118 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5bd1dc37-2b68-44e9-8b82-d836ab18a7a0" containerName="revoke-keys" Oct 30 13:07:40.876175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.876133 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd1dc37-2b68-44e9-8b82-d836ab18a7a0" containerName="revoke-keys" Oct 30 13:07:40.876258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.876249 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd1dc37-2b68-44e9-8b82-d836ab18a7a0" containerName="revoke-keys" Oct 30 13:07:40.880947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.880926 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:40.911717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.909913 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:07:40.911717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.909973 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:40.915742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.915711 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26"] Oct 30 13:07:40.999841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.999801 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb60638f-4qm26\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:41.000045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.999921 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb60638f-4qm26\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:41.000045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:40.999979 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwz4c\" (UniqueName: \"kubernetes.io/projected/5f926f34-337e-4cd1-a1af-b0c443c154cb-kube-api-access-bwz4c\") pod \"maas-api-revoke-sub-bb60638f-4qm26\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:41.100749 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:41.100713 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwz4c\" (UniqueName: \"kubernetes.io/projected/5f926f34-337e-4cd1-a1af-b0c443c154cb-kube-api-access-bwz4c\") pod \"maas-api-revoke-sub-bb60638f-4qm26\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:41.100943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:41.100847 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb60638f-4qm26\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:41.100943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:41.100905 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb60638f-4qm26\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:41.101634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:41.101609 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb60638f-4qm26\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:41.103506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:41.103480 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb60638f-4qm26\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:41.124810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:41.124785 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwz4c\" (UniqueName: \"kubernetes.io/projected/5f926f34-337e-4cd1-a1af-b0c443c154cb-kube-api-access-bwz4c\") pod \"maas-api-revoke-sub-bb60638f-4qm26\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:41.194584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:41.194507 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:41.386944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:41.386909 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:41.518230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:41.518201 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26"] Oct 30 13:07:41.520753 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:07:41.520723 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f926f34_337e_4cd1_a1af_b0c443c154cb.slice/crio-bbb0306562b65c1e8cded6f643e2679e73fa528a16d6b0788b55d1b27a8596bc WatchSource:0}: Error finding container bbb0306562b65c1e8cded6f643e2679e73fa528a16d6b0788b55d1b27a8596bc: Status 404 returned error can't find the container with id bbb0306562b65c1e8cded6f643e2679e73fa528a16d6b0788b55d1b27a8596bc Oct 30 13:07:41.619902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:41.619869 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" event={"ID":"5f926f34-337e-4cd1-a1af-b0c443c154cb","Type":"ContainerStarted","Data":"bbb0306562b65c1e8cded6f643e2679e73fa528a16d6b0788b55d1b27a8596bc"} Oct 30 13:07:42.629142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:42.629106 5049 generic.go:356] "Generic (PLEG): container finished" podID="5f926f34-337e-4cd1-a1af-b0c443c154cb" containerID="b5fc6ca86221f359764e87297adad8645f86a3b8b9750360341fcace12e7f1cd" exitCode=0 Oct 30 13:07:42.629540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:42.629186 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" event={"ID":"5f926f34-337e-4cd1-a1af-b0c443c154cb","Type":"ContainerDied","Data":"b5fc6ca86221f359764e87297adad8645f86a3b8b9750360341fcace12e7f1cd"} Oct 30 13:07:42.791353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:42.791261 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6"] Oct 30 13:07:42.800616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:42.800588 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:42.820565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:42.820532 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6"] Oct 30 13:07:42.924472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:42.924433 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqpw\" (UniqueName: \"kubernetes.io/projected/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-kube-api-access-jpqpw\") pod \"maas-api-revoke-sub-857e706d-sdzv6\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:42.924646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:42.924622 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-857e706d-sdzv6\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:42.924761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:42.924743 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-857e706d-sdzv6\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:43.026283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.026247 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-857e706d-sdzv6\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:43.026490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.026357 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-857e706d-sdzv6\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:43.026490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.026438 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqpw\" (UniqueName: \"kubernetes.io/projected/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-kube-api-access-jpqpw\") pod \"maas-api-revoke-sub-857e706d-sdzv6\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:43.027138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.027114 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-857e706d-sdzv6\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:43.028690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.028669 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-857e706d-sdzv6\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:43.050309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.050285 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqpw\" (UniqueName: \"kubernetes.io/projected/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-kube-api-access-jpqpw\") pod \"maas-api-revoke-sub-857e706d-sdzv6\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:43.120372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.120339 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:43.345622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.345509 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:43.464679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.464638 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6"] Oct 30 13:07:43.475288 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:07:43.475246 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88d82c2_c6c9_4568_9ed8_921a7bfc7103.slice/crio-2f1bbcc5d5309f375f3b2b5ac04a9cf8ba0757a84415625c5c62cb022e1baf2b WatchSource:0}: Error finding container 2f1bbcc5d5309f375f3b2b5ac04a9cf8ba0757a84415625c5c62cb022e1baf2b: Status 404 returned error can't find the container with id 2f1bbcc5d5309f375f3b2b5ac04a9cf8ba0757a84415625c5c62cb022e1baf2b Oct 30 13:07:43.638267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.638179 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" event={"ID":"f88d82c2-c6c9-4568-9ed8-921a7bfc7103","Type":"ContainerStarted","Data":"2f1bbcc5d5309f375f3b2b5ac04a9cf8ba0757a84415625c5c62cb022e1baf2b"} Oct 30 13:07:43.808307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.808277 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:43.837055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.837030 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd"] Oct 30 13:07:43.837826 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.837806 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5f926f34-337e-4cd1-a1af-b0c443c154cb" containerName="revoke-keys" Oct 30 13:07:43.837959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.837829 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f926f34-337e-4cd1-a1af-b0c443c154cb" containerName="revoke-keys" Oct 30 13:07:43.838001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.837980 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f926f34-337e-4cd1-a1af-b0c443c154cb" containerName="revoke-keys" Oct 30 13:07:43.841919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.841901 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:43.864480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.864445 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd"] Oct 30 13:07:43.946854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.946762 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-xks-ca\") pod \"5f926f34-337e-4cd1-a1af-b0c443c154cb\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " Oct 30 13:07:43.947024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.946888 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-openshift-ca\") pod \"5f926f34-337e-4cd1-a1af-b0c443c154cb\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " Oct 30 13:07:43.947024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.946981 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f926f34-337e-4cd1-a1af-b0c443c154cb-kube-api-access-bwz4c\" (UniqueName: \"kubernetes.io/projected/5f926f34-337e-4cd1-a1af-b0c443c154cb-kube-api-access-bwz4c\") pod \"5f926f34-337e-4cd1-a1af-b0c443c154cb\" (UID: \"5f926f34-337e-4cd1-a1af-b0c443c154cb\") " Oct 30 13:07:43.947242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.947218 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-openshift-ca" pod "5f926f34-337e-4cd1-a1af-b0c443c154cb" (UID: "5f926f34-337e-4cd1-a1af-b0c443c154cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:43.947299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.947265 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a838a82-7ssnd\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:43.947400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.947384 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh7db\" (UniqueName: \"kubernetes.io/projected/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-kube-api-access-nh7db\") pod \"maas-api-revoke-sub-0a838a82-7ssnd\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:43.947571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.947542 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a838a82-7ssnd\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:43.947763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.947739 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:43.949115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.949095 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-xks-ca" pod "5f926f34-337e-4cd1-a1af-b0c443c154cb" (UID: "5f926f34-337e-4cd1-a1af-b0c443c154cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:43.949209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:43.949126 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f926f34-337e-4cd1-a1af-b0c443c154cb-kube-api-access-bwz4c" pod "5f926f34-337e-4cd1-a1af-b0c443c154cb" (UID: "5f926f34-337e-4cd1-a1af-b0c443c154cb"). InnerVolumeSpecName "kube-api-access-bwz4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:44.045258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.045227 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb"] Oct 30 13:07:44.048792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.048761 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a838a82-7ssnd\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:44.048932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.048843 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh7db\" (UniqueName: \"kubernetes.io/projected/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-kube-api-access-nh7db\") pod \"maas-api-revoke-sub-0a838a82-7ssnd\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:44.049026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.048997 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a838a82-7ssnd\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:44.049141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.049125 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwz4c\" (UniqueName: \"kubernetes.io/projected/5f926f34-337e-4cd1-a1af-b0c443c154cb-kube-api-access-bwz4c\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:44.049216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.049148 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f926f34-337e-4cd1-a1af-b0c443c154cb-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:44.049216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.049201 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.049509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.049478 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a838a82-7ssnd\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:44.051469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.051442 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a838a82-7ssnd\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:44.073025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.071415 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb"] Oct 30 13:07:44.074642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.074616 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh7db\" (UniqueName: \"kubernetes.io/projected/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-kube-api-access-nh7db\") pod \"maas-api-revoke-sub-0a838a82-7ssnd\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:44.150170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.150141 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h647q\" (UniqueName: \"kubernetes.io/projected/a31fc1d8-9c82-4ae6-b58b-9843e438e997-kube-api-access-h647q\") pod \"maas-api-revoke-sub-0bc60d35-sb2kb\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.150337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.150193 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc60d35-sb2kb\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.150394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.150363 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc60d35-sb2kb\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.157784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.157762 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:44.251516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.251483 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h647q\" (UniqueName: \"kubernetes.io/projected/a31fc1d8-9c82-4ae6-b58b-9843e438e997-kube-api-access-h647q\") pod \"maas-api-revoke-sub-0bc60d35-sb2kb\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.251695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.251536 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc60d35-sb2kb\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.251695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.251584 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc60d35-sb2kb\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.252707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.252678 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc60d35-sb2kb\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.254032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.254004 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc60d35-sb2kb\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.282208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.282180 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h647q\" (UniqueName: \"kubernetes.io/projected/a31fc1d8-9c82-4ae6-b58b-9843e438e997-kube-api-access-h647q\") pod \"maas-api-revoke-sub-0bc60d35-sb2kb\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.364380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.364343 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:44.480011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.479973 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd"] Oct 30 13:07:44.486531 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:07:44.486502 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4d7f8b_dc72_4b78_ba2b_8505a18eaf71.slice/crio-15e8b7f6fc01e940eb788a8ec740b6bf286a07095d37df5d4de11294f525038e WatchSource:0}: Error finding container 15e8b7f6fc01e940eb788a8ec740b6bf286a07095d37df5d4de11294f525038e: Status 404 returned error can't find the container with id 15e8b7f6fc01e940eb788a8ec740b6bf286a07095d37df5d4de11294f525038e Oct 30 13:07:44.647703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.647663 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" event={"ID":"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71","Type":"ContainerStarted","Data":"15e8b7f6fc01e940eb788a8ec740b6bf286a07095d37df5d4de11294f525038e"} Oct 30 13:07:44.649411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.649383 5049 generic.go:356] "Generic (PLEG): container finished" podID="f88d82c2-c6c9-4568-9ed8-921a7bfc7103" containerID="b78cf5ad28b92e6fc91d6e398dfae8a4c5f1b08c10548852564b7515799a364c" exitCode=0 Oct 30 13:07:44.649545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.649474 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" event={"ID":"f88d82c2-c6c9-4568-9ed8-921a7bfc7103","Type":"ContainerDied","Data":"b78cf5ad28b92e6fc91d6e398dfae8a4c5f1b08c10548852564b7515799a364c"} Oct 30 13:07:44.650906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.650888 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" Oct 30 13:07:44.651006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.650939 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26" event={"ID":"5f926f34-337e-4cd1-a1af-b0c443c154cb","Type":"ContainerDied","Data":"bbb0306562b65c1e8cded6f643e2679e73fa528a16d6b0788b55d1b27a8596bc"} Oct 30 13:07:44.651006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.650991 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbb0306562b65c1e8cded6f643e2679e73fa528a16d6b0788b55d1b27a8596bc" Oct 30 13:07:44.674568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:44.674542 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb"] Oct 30 13:07:44.675429 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:07:44.675406 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31fc1d8_9c82_4ae6_b58b_9843e438e997.slice/crio-bdb54cfeb12ecc560056fb514f15df86e264fae9696ce479923eac56585bc31b WatchSource:0}: Error finding container bdb54cfeb12ecc560056fb514f15df86e264fae9696ce479923eac56585bc31b: Status 404 returned error can't find the container with id bdb54cfeb12ecc560056fb514f15df86e264fae9696ce479923eac56585bc31b Oct 30 13:07:45.199488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.198671 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:45.660991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.660955 5049 generic.go:356] "Generic (PLEG): container finished" podID="fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71" containerID="308e0a7468dc933c6a27db1df5de541069d101899c339687de612c080e140fe1" exitCode=0 Oct 30 13:07:45.661430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.661040 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" event={"ID":"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71","Type":"ContainerDied","Data":"308e0a7468dc933c6a27db1df5de541069d101899c339687de612c080e140fe1"} Oct 30 13:07:45.663021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.662992 5049 generic.go:356] "Generic (PLEG): container finished" podID="a31fc1d8-9c82-4ae6-b58b-9843e438e997" containerID="8ff0ffdebe9c72fcadd274cb85c059e4878ac5d353379e2496d7bd1a348c87ab" exitCode=0 Oct 30 13:07:45.663179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.663078 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" event={"ID":"a31fc1d8-9c82-4ae6-b58b-9843e438e997","Type":"ContainerDied","Data":"8ff0ffdebe9c72fcadd274cb85c059e4878ac5d353379e2496d7bd1a348c87ab"} Oct 30 13:07:45.663179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.663138 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" event={"ID":"a31fc1d8-9c82-4ae6-b58b-9843e438e997","Type":"ContainerStarted","Data":"bdb54cfeb12ecc560056fb514f15df86e264fae9696ce479923eac56585bc31b"} Oct 30 13:07:45.825709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.825680 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:45.875382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.872192 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-openshift-ca\") pod \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " Oct 30 13:07:45.875382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.872274 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-kube-api-access-jpqpw\" (UniqueName: \"kubernetes.io/projected/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-kube-api-access-jpqpw\") pod \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " Oct 30 13:07:45.875382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.872414 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-xks-ca\") pod \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\" (UID: \"f88d82c2-c6c9-4568-9ed8-921a7bfc7103\") " Oct 30 13:07:45.876132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.876096 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-openshift-ca" pod "f88d82c2-c6c9-4568-9ed8-921a7bfc7103" (UID: "f88d82c2-c6c9-4568-9ed8-921a7bfc7103"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:45.878159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.878130 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-kube-api-access-jpqpw" pod "f88d82c2-c6c9-4568-9ed8-921a7bfc7103" (UID: "f88d82c2-c6c9-4568-9ed8-921a7bfc7103"). InnerVolumeSpecName "kube-api-access-jpqpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:45.878446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.878423 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-xks-ca" pod "f88d82c2-c6c9-4568-9ed8-921a7bfc7103" (UID: "f88d82c2-c6c9-4568-9ed8-921a7bfc7103"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:45.908444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.908411 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:45.973871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.973838 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:45.973871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.973866 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpqpw\" (UniqueName: \"kubernetes.io/projected/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-kube-api-access-jpqpw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:45.973871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:45.973877 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88d82c2-c6c9-4568-9ed8-921a7bfc7103-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:46.679466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.679435 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" Oct 30 13:07:46.679466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.679445 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6" event={"ID":"f88d82c2-c6c9-4568-9ed8-921a7bfc7103","Type":"ContainerDied","Data":"2f1bbcc5d5309f375f3b2b5ac04a9cf8ba0757a84415625c5c62cb022e1baf2b"} Oct 30 13:07:46.679912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.679527 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f1bbcc5d5309f375f3b2b5ac04a9cf8ba0757a84415625c5c62cb022e1baf2b" Oct 30 13:07:46.867063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.867032 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:46.883415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.883388 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-xks-ca\") pod \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " Oct 30 13:07:46.883540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.883455 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-openshift-ca\") pod \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " Oct 30 13:07:46.883540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.883486 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-kube-api-access-nh7db\" (UniqueName: \"kubernetes.io/projected/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-kube-api-access-nh7db\") pod \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\" (UID: \"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71\") " Oct 30 13:07:46.883891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.883863 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-openshift-ca" pod "fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71" (UID: "fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:46.884017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.883995 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:46.885682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.885652 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-kube-api-access-nh7db" pod "fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71" (UID: "fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71"). InnerVolumeSpecName "kube-api-access-nh7db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:46.885786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.885660 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-xks-ca" pod "fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71" (UID: "fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:46.899765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.899742 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:46.985254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.985169 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-openshift-ca\") pod \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " Oct 30 13:07:46.985254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.985219 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-xks-ca\") pod \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " Oct 30 13:07:46.985477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.985306 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31fc1d8-9c82-4ae6-b58b-9843e438e997-kube-api-access-h647q\" (UniqueName: \"kubernetes.io/projected/a31fc1d8-9c82-4ae6-b58b-9843e438e997-kube-api-access-h647q\") pod \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\" (UID: \"a31fc1d8-9c82-4ae6-b58b-9843e438e997\") " Oct 30 13:07:46.985621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.985596 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-openshift-ca" pod "a31fc1d8-9c82-4ae6-b58b-9843e438e997" (UID: "a31fc1d8-9c82-4ae6-b58b-9843e438e997"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:46.985867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.985847 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:46.985930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.985874 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:46.985930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.985890 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh7db\" (UniqueName: \"kubernetes.io/projected/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71-kube-api-access-nh7db\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:46.987412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.987388 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-xks-ca" pod "a31fc1d8-9c82-4ae6-b58b-9843e438e997" (UID: "a31fc1d8-9c82-4ae6-b58b-9843e438e997"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:46.987653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:46.987625 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31fc1d8-9c82-4ae6-b58b-9843e438e997-kube-api-access-h647q" pod "a31fc1d8-9c82-4ae6-b58b-9843e438e997" (UID: "a31fc1d8-9c82-4ae6-b58b-9843e438e997"). InnerVolumeSpecName "kube-api-access-h647q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:47.086869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:47.086839 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31fc1d8-9c82-4ae6-b58b-9843e438e997-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:47.086869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:47.086866 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h647q\" (UniqueName: \"kubernetes.io/projected/a31fc1d8-9c82-4ae6-b58b-9843e438e997-kube-api-access-h647q\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:47.172007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:47.171959 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:47.687899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:47.687806 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" Oct 30 13:07:47.687899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:47.687821 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd" event={"ID":"fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71","Type":"ContainerDied","Data":"15e8b7f6fc01e940eb788a8ec740b6bf286a07095d37df5d4de11294f525038e"} Oct 30 13:07:47.687899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:47.687875 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15e8b7f6fc01e940eb788a8ec740b6bf286a07095d37df5d4de11294f525038e" Oct 30 13:07:47.689458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:47.689433 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" Oct 30 13:07:47.689556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:47.689480 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb" event={"ID":"a31fc1d8-9c82-4ae6-b58b-9843e438e997","Type":"ContainerDied","Data":"bdb54cfeb12ecc560056fb514f15df86e264fae9696ce479923eac56585bc31b"} Oct 30 13:07:47.689556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:47.689524 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb54cfeb12ecc560056fb514f15df86e264fae9696ce479923eac56585bc31b" Oct 30 13:07:48.922864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:48.922826 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:50.719755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:50.719713 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:52.913608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.913564 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:52.988028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.987993 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk"] Oct 30 13:07:52.988581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.988566 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f88d82c2-c6c9-4568-9ed8-921a7bfc7103" containerName="revoke-keys" Oct 30 13:07:52.988639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.988585 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f88d82c2-c6c9-4568-9ed8-921a7bfc7103" containerName="revoke-keys" Oct 30 13:07:52.988639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.988600 5049 cpu_manager.go:414] "containerMap: removing container" podUID="fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71" containerName="revoke-keys" Oct 30 13:07:52.988639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.988605 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71" containerName="revoke-keys" Oct 30 13:07:52.988639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.988631 5049 cpu_manager.go:414] "containerMap: removing container" podUID="a31fc1d8-9c82-4ae6-b58b-9843e438e997" containerName="revoke-keys" Oct 30 13:07:52.988639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.988636 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31fc1d8-9c82-4ae6-b58b-9843e438e997" containerName="revoke-keys" Oct 30 13:07:52.988792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.988734 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31fc1d8-9c82-4ae6-b58b-9843e438e997" containerName="revoke-keys" Oct 30 13:07:52.988792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.988745 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f88d82c2-c6c9-4568-9ed8-921a7bfc7103" containerName="revoke-keys" Oct 30 13:07:52.988792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.988753 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71" containerName="revoke-keys" Oct 30 13:07:52.992304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:52.992287 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.008232 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.008209 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:07:53.026710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.026677 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk"] Oct 30 13:07:53.027427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.027400 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d47a50f0-mfjgk\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.027569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.027439 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d47a50f0-mfjgk\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.027703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.027677 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv5qd\" (UniqueName: \"kubernetes.io/projected/31f1ee5b-0838-4554-9db8-3209694fbcb6-kube-api-access-xv5qd\") pod \"maas-api-revoke-sub-d47a50f0-mfjgk\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.040013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.039889 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:53.128801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.128769 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d47a50f0-mfjgk\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.128801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.128805 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d47a50f0-mfjgk\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.129028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.128886 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv5qd\" (UniqueName: \"kubernetes.io/projected/31f1ee5b-0838-4554-9db8-3209694fbcb6-kube-api-access-xv5qd\") pod \"maas-api-revoke-sub-d47a50f0-mfjgk\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.129769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.129701 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d47a50f0-mfjgk\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.131374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.131349 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d47a50f0-mfjgk\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.151930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.151902 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv5qd\" (UniqueName: \"kubernetes.io/projected/31f1ee5b-0838-4554-9db8-3209694fbcb6-kube-api-access-xv5qd\") pod \"maas-api-revoke-sub-d47a50f0-mfjgk\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.306046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.306011 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:53.565732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.565644 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6"] Oct 30 13:07:53.570334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.570294 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.598811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.598709 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6"] Oct 30 13:07:53.635585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.635526 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.635773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.635753 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7jwh\" (UniqueName: \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-kube-api-access-f7jwh\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.635891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.635877 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.636001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.635987 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd729fe-55e6-46d3-bc51-ddac03b27463-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.636155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.636140 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.636400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.636361 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.636552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.636537 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.636717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.636703 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.636823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.636808 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.640542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.639609 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk"] Oct 30 13:07:53.644759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.644736 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf"] Oct 30 13:07:53.654445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.654424 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.681433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.681405 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf"] Oct 30 13:07:53.737720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.737688 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxftp\" (UniqueName: \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-kube-api-access-gxftp\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.737899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.737731 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.737899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.737807 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.737899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.737866 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.738076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.737941 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.738076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.738001 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.738076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.738035 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.738076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.738072 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.738286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.738121 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.738286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.738185 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.738286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.738210 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.738286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.738218 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae7e80-2c37-47f7-8540-791bfbf7011d-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.738602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.738580 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.738785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.738768 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jwh\" (UniqueName: \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-kube-api-access-f7jwh\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.738913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.738895 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.739037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.739018 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.739165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.739150 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.739266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.739250 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.739392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.739274 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd729fe-55e6-46d3-bc51-ddac03b27463-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.739564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.739538 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.739945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.739924 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd729fe-55e6-46d3-bc51-ddac03b27463-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.740095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.740077 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.740194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.740163 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.771030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.771005 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" event={"ID":"31f1ee5b-0838-4554-9db8-3209694fbcb6","Type":"ContainerStarted","Data":"51d8d38516d8edcecfce6066ef76e36d55fa0504e6c80c1e1f38ed2928c2da91"} Oct 30 13:07:53.806994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.806963 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.807184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.807081 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.807184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.807108 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.807969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.807947 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7jwh\" (UniqueName: \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-kube-api-access-f7jwh\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:53.840569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.840546 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.840675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.840597 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.840675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.840620 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae7e80-2c37-47f7-8540-791bfbf7011d-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.840675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.840657 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.840847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.840680 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.840847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.840760 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxftp\" (UniqueName: \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-kube-api-access-gxftp\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.840847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.840801 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.841009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.840895 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.841009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.840967 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.841118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.841016 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.841118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.840964 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.841212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.841157 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.841212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.841199 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.841899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.841874 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae7e80-2c37-47f7-8540-791bfbf7011d-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.843229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.843211 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.843953 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.843934 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.864831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.864809 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.864921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.864896 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxftp\" (UniqueName: \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-kube-api-access-gxftp\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:53.884757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:53.884738 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:54.036483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.036450 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:54.284356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.284314 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6"] Oct 30 13:07:54.284941 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:07:54.284912 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd729fe_55e6_46d3_bc51_ddac03b27463.slice/crio-1d7206d60af51b1701956182d04a22a2fa48347e6a68b0011b4e8fba27872217 WatchSource:0}: Error finding container 1d7206d60af51b1701956182d04a22a2fa48347e6a68b0011b4e8fba27872217: Status 404 returned error can't find the container with id 1d7206d60af51b1701956182d04a22a2fa48347e6a68b0011b4e8fba27872217 Oct 30 13:07:54.287525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.287496 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.287650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.287577 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.287650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.287624 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.359380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.358756 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf"] Oct 30 13:07:54.362640 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:07:54.362614 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcae7e80_2c37_47f7_8540_791bfbf7011d.slice/crio-f11def5fc3e181b708e519a92b74b1bfa3fb36e0368927d2d166629f64984c5e WatchSource:0}: Error finding container f11def5fc3e181b708e519a92b74b1bfa3fb36e0368927d2d166629f64984c5e: Status 404 returned error can't find the container with id f11def5fc3e181b708e519a92b74b1bfa3fb36e0368927d2d166629f64984c5e Oct 30 13:07:54.364677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.364648 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.364779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.364710 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.364779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.364754 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.365893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.365863 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955"] Oct 30 13:07:54.371573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.371547 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.388279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.388251 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955"] Oct 30 13:07:54.455694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.455658 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-884fa692-9k955\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.455901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.455876 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwzz4\" (UniqueName: \"kubernetes.io/projected/c0536a8a-196f-4607-bf42-84e47e979236-kube-api-access-hwzz4\") pod \"maas-api-revoke-sub-884fa692-9k955\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.455996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.455978 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-884fa692-9k955\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.558078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.558039 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-884fa692-9k955\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.558250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.558116 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzz4\" (UniqueName: \"kubernetes.io/projected/c0536a8a-196f-4607-bf42-84e47e979236-kube-api-access-hwzz4\") pod \"maas-api-revoke-sub-884fa692-9k955\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.558250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.558156 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-884fa692-9k955\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.558973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.558944 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-884fa692-9k955\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.561346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.561302 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-884fa692-9k955\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.583372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.583344 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwzz4\" (UniqueName: \"kubernetes.io/projected/c0536a8a-196f-4607-bf42-84e47e979236-kube-api-access-hwzz4\") pod \"maas-api-revoke-sub-884fa692-9k955\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.734390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.734361 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:54.778763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.778518 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:54.786538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.786504 5049 generic.go:356] "Generic (PLEG): container finished" podID="31f1ee5b-0838-4554-9db8-3209694fbcb6" containerID="40986a1e1d85c43180f9021597a2687db026652b98dcf4ac29edf8ea39fa1355" exitCode=0 Oct 30 13:07:54.786686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.786621 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" event={"ID":"31f1ee5b-0838-4554-9db8-3209694fbcb6","Type":"ContainerDied","Data":"40986a1e1d85c43180f9021597a2687db026652b98dcf4ac29edf8ea39fa1355"} Oct 30 13:07:54.788524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.788492 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" event={"ID":"6bd729fe-55e6-46d3-bc51-ddac03b27463","Type":"ContainerStarted","Data":"888d95f44fbb9e2a8dd23ffaa58fa512f8f2ef512cf35845e0053160bef0b223"} Oct 30 13:07:54.788661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.788552 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" event={"ID":"6bd729fe-55e6-46d3-bc51-ddac03b27463","Type":"ContainerStarted","Data":"1d7206d60af51b1701956182d04a22a2fa48347e6a68b0011b4e8fba27872217"} Oct 30 13:07:54.790364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.790311 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" event={"ID":"dcae7e80-2c37-47f7-8540-791bfbf7011d","Type":"ContainerStarted","Data":"20cf340d2aa391cf64324c79cba7eb6274ed53ccd291feebd7360bf5e1f3d428"} Oct 30 13:07:54.790480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.790384 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" event={"ID":"dcae7e80-2c37-47f7-8540-791bfbf7011d","Type":"ContainerStarted","Data":"f11def5fc3e181b708e519a92b74b1bfa3fb36e0368927d2d166629f64984c5e"} Oct 30 13:07:54.886704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.886666 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:07:54.889603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:54.889540 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" podStartSLOduration=1.8895227110000001 podStartE2EDuration="1.889522711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:54.881455175 +0000 UTC m=+981.696649624" watchObservedRunningTime="2026-10-30 13:07:54.889522711 +0000 UTC m=+981.704717147" Oct 30 13:07:55.043458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:55.043355 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:07:55.044044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:55.043992 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:07:55.051571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:55.051520 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:07:55.072028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:55.071977 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" podStartSLOduration=2.07196178 podStartE2EDuration="2.07196178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:54.942867833 +0000 UTC m=+981.758062257" watchObservedRunningTime="2026-10-30 13:07:55.07196178 +0000 UTC m=+981.887156202" Oct 30 13:07:55.079380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:55.077314 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955"] Oct 30 13:07:55.801174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:55.801138 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" event={"ID":"c0536a8a-196f-4607-bf42-84e47e979236","Type":"ContainerStarted","Data":"baa8d5ad2547552a982b60a905f26701a66b7715f9895cf6486d47d93bff77c1"} Oct 30 13:07:55.801374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:55.801191 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" event={"ID":"c0536a8a-196f-4607-bf42-84e47e979236","Type":"ContainerStarted","Data":"376943864e5ff28d270b8fe5a76c8dbad4fc84e443871a8ecc783a94759ddb13"} Oct 30 13:07:55.886519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:55.886484 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:07:55.886844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:55.886539 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:07:56.076401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.076273 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:07:56.076830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.076393 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:07:56.278486 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.278450 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:56.289726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.289688 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-openshift-ca\") pod \"31f1ee5b-0838-4554-9db8-3209694fbcb6\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " Oct 30 13:07:56.289959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.289946 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f1ee5b-0838-4554-9db8-3209694fbcb6-kube-api-access-xv5qd\" (UniqueName: \"kubernetes.io/projected/31f1ee5b-0838-4554-9db8-3209694fbcb6-kube-api-access-xv5qd\") pod \"31f1ee5b-0838-4554-9db8-3209694fbcb6\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " Oct 30 13:07:56.290118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.289983 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-openshift-ca" pod "31f1ee5b-0838-4554-9db8-3209694fbcb6" (UID: "31f1ee5b-0838-4554-9db8-3209694fbcb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:56.290524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.290509 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.291858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.291837 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f1ee5b-0838-4554-9db8-3209694fbcb6-kube-api-access-xv5qd" pod "31f1ee5b-0838-4554-9db8-3209694fbcb6" (UID: "31f1ee5b-0838-4554-9db8-3209694fbcb6"). InnerVolumeSpecName "kube-api-access-xv5qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:56.336396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.336064 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" podStartSLOduration=2.336041988 podStartE2EDuration="2.336041988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:55.86152172 +0000 UTC m=+982.676716144" watchObservedRunningTime="2026-10-30 13:07:56.336041988 +0000 UTC m=+983.151236414" Oct 30 13:07:56.390314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.390279 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7"] Oct 30 13:07:56.391023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.391005 5049 cpu_manager.go:414] "containerMap: removing container" podUID="31f1ee5b-0838-4554-9db8-3209694fbcb6" containerName="revoke-keys" Oct 30 13:07:56.391065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.391024 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f1ee5b-0838-4554-9db8-3209694fbcb6" containerName="revoke-keys" Oct 30 13:07:56.391100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.391062 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-xks-ca\") pod \"31f1ee5b-0838-4554-9db8-3209694fbcb6\" (UID: \"31f1ee5b-0838-4554-9db8-3209694fbcb6\") " Oct 30 13:07:56.391160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.391150 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f1ee5b-0838-4554-9db8-3209694fbcb6" containerName="revoke-keys" Oct 30 13:07:56.391691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.391674 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv5qd\" (UniqueName: \"kubernetes.io/projected/31f1ee5b-0838-4554-9db8-3209694fbcb6-kube-api-access-xv5qd\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.393677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.393648 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-xks-ca" pod "31f1ee5b-0838-4554-9db8-3209694fbcb6" (UID: "31f1ee5b-0838-4554-9db8-3209694fbcb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:56.394861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.394845 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.466382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.425273 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7"] Oct 30 13:07:56.492713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.492674 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73bcaeed-tzwn7\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.493850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.493622 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73bcaeed-tzwn7\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.493850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.493668 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5sp\" (UniqueName: \"kubernetes.io/projected/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-kube-api-access-kk5sp\") pod \"maas-api-revoke-sub-73bcaeed-tzwn7\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.493850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.493821 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f1ee5b-0838-4554-9db8-3209694fbcb6-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:56.594503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.594409 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73bcaeed-tzwn7\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.594503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.594448 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73bcaeed-tzwn7\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.594503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.594480 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5sp\" (UniqueName: \"kubernetes.io/projected/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-kube-api-access-kk5sp\") pod \"maas-api-revoke-sub-73bcaeed-tzwn7\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.595351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.595311 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73bcaeed-tzwn7\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.597089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.597066 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73bcaeed-tzwn7\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.654175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.644152 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5sp\" (UniqueName: \"kubernetes.io/projected/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-kube-api-access-kk5sp\") pod \"maas-api-revoke-sub-73bcaeed-tzwn7\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.695662 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:07:56.695587 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:07:56.794516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.794476 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:07:56.886538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.886462 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:07:56.886822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.886793 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:07:56.889164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.888276 5049 generic.go:356] "Generic (PLEG): container finished" podID="c0536a8a-196f-4607-bf42-84e47e979236" containerID="baa8d5ad2547552a982b60a905f26701a66b7715f9895cf6486d47d93bff77c1" exitCode=0 Oct 30 13:07:56.889164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.888311 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" event={"ID":"c0536a8a-196f-4607-bf42-84e47e979236","Type":"ContainerDied","Data":"baa8d5ad2547552a982b60a905f26701a66b7715f9895cf6486d47d93bff77c1"} Oct 30 13:07:56.892370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.892278 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" Oct 30 13:07:56.892509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.892371 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk" event={"ID":"31f1ee5b-0838-4554-9db8-3209694fbcb6","Type":"ContainerDied","Data":"51d8d38516d8edcecfce6066ef76e36d55fa0504e6c80c1e1f38ed2928c2da91"} Oct 30 13:07:56.892509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.892425 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51d8d38516d8edcecfce6066ef76e36d55fa0504e6c80c1e1f38ed2928c2da91" Oct 30 13:07:56.993618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:56.993570 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:57.070386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:57.067742 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:07:57.070386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:57.067803 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:07:57.276880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:57.276689 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7"] Oct 30 13:07:57.885339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:57.885238 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:07:57.885339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:57.885302 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:07:57.901657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:57.901615 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" event={"ID":"80e20f8a-c8b2-4563-a0f9-25afdc0cea37","Type":"ContainerStarted","Data":"6154cc8f60e98791ec735ba28fabf7131aa984bfb96e21b963b7b795a916b053"} Oct 30 13:07:57.901841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:57.901668 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" event={"ID":"80e20f8a-c8b2-4563-a0f9-25afdc0cea37","Type":"ContainerStarted","Data":"cdc4a6ae6540d79f12d97ffe76c6ee79e6f88d5fe45055d16ab1802dcbf36ec1"} Oct 30 13:07:57.970235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:57.969585 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" podStartSLOduration=1.969561574 podStartE2EDuration="1.969561574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:57.962624025 +0000 UTC m=+984.777818448" watchObservedRunningTime="2026-10-30 13:07:57.969561574 +0000 UTC m=+984.784756000" Oct 30 13:07:58.053952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.053919 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:07:58.054828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.054189 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:07:58.277351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.276807 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:58.480137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.480099 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-xks-ca\") pod \"c0536a8a-196f-4607-bf42-84e47e979236\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " Oct 30 13:07:58.480301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.480199 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-openshift-ca\") pod \"c0536a8a-196f-4607-bf42-84e47e979236\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " Oct 30 13:07:58.480643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.480616 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0536a8a-196f-4607-bf42-84e47e979236-kube-api-access-hwzz4\" (UniqueName: \"kubernetes.io/projected/c0536a8a-196f-4607-bf42-84e47e979236-kube-api-access-hwzz4\") pod \"c0536a8a-196f-4607-bf42-84e47e979236\" (UID: \"c0536a8a-196f-4607-bf42-84e47e979236\") " Oct 30 13:07:58.480804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.480770 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-openshift-ca" pod "c0536a8a-196f-4607-bf42-84e47e979236" (UID: "c0536a8a-196f-4607-bf42-84e47e979236"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:58.481288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.481261 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:58.482772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.482746 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-xks-ca" pod "c0536a8a-196f-4607-bf42-84e47e979236" (UID: "c0536a8a-196f-4607-bf42-84e47e979236"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:58.483362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.483339 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0536a8a-196f-4607-bf42-84e47e979236-kube-api-access-hwzz4" pod "c0536a8a-196f-4607-bf42-84e47e979236" (UID: "c0536a8a-196f-4607-bf42-84e47e979236"). InnerVolumeSpecName "kube-api-access-hwzz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:58.582490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.582456 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0536a8a-196f-4607-bf42-84e47e979236-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:58.582490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.582486 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwzz4\" (UniqueName: \"kubernetes.io/projected/c0536a8a-196f-4607-bf42-84e47e979236-kube-api-access-hwzz4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:07:58.885807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.885771 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:07:58.885989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.885821 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:07:58.945346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.937125 5049 generic.go:356] "Generic (PLEG): container finished" podID="80e20f8a-c8b2-4563-a0f9-25afdc0cea37" containerID="6154cc8f60e98791ec735ba28fabf7131aa984bfb96e21b963b7b795a916b053" exitCode=0 Oct 30 13:07:58.945346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.940535 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" event={"ID":"80e20f8a-c8b2-4563-a0f9-25afdc0cea37","Type":"ContainerDied","Data":"6154cc8f60e98791ec735ba28fabf7131aa984bfb96e21b963b7b795a916b053"} Oct 30 13:07:58.961363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.960732 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" event={"ID":"c0536a8a-196f-4607-bf42-84e47e979236","Type":"ContainerDied","Data":"376943864e5ff28d270b8fe5a76c8dbad4fc84e443871a8ecc783a94759ddb13"} Oct 30 13:07:58.961363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.960795 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="376943864e5ff28d270b8fe5a76c8dbad4fc84e443871a8ecc783a94759ddb13" Oct 30 13:07:58.961363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:58.960862 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955" Oct 30 13:07:59.039752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:59.039703 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:07:59.040021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:59.039990 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:07:59.159595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:59.159495 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:59.885455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:59.885417 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:07:59.885848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:07:59.885470 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:08:00.043349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.042886 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:08:00.043349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.042952 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:08:00.328369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.314842 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:08:00.464584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.462538 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-xks-ca\") pod \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " Oct 30 13:08:00.464584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.462660 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-openshift-ca\") pod \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " Oct 30 13:08:00.464584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.462779 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-kube-api-access-kk5sp\" (UniqueName: \"kubernetes.io/projected/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-kube-api-access-kk5sp\") pod \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\" (UID: \"80e20f8a-c8b2-4563-a0f9-25afdc0cea37\") " Oct 30 13:08:00.468204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.467724 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-openshift-ca" pod "80e20f8a-c8b2-4563-a0f9-25afdc0cea37" (UID: "80e20f8a-c8b2-4563-a0f9-25afdc0cea37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:00.468790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.468763 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-kube-api-access-kk5sp" pod "80e20f8a-c8b2-4563-a0f9-25afdc0cea37" (UID: "80e20f8a-c8b2-4563-a0f9-25afdc0cea37"). InnerVolumeSpecName "kube-api-access-kk5sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:00.469096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.469073 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-xks-ca" pod "80e20f8a-c8b2-4563-a0f9-25afdc0cea37" (UID: "80e20f8a-c8b2-4563-a0f9-25afdc0cea37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:00.564271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.564230 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:00.564271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.564266 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk5sp\" (UniqueName: \"kubernetes.io/projected/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-kube-api-access-kk5sp\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:00.564271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.564279 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e20f8a-c8b2-4563-a0f9-25afdc0cea37-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:00.885696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.885600 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:08:00.885696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.885666 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 13:08:00.986826 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.986786 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" event={"ID":"80e20f8a-c8b2-4563-a0f9-25afdc0cea37","Type":"ContainerDied","Data":"cdc4a6ae6540d79f12d97ffe76c6ee79e6f88d5fe45055d16ab1802dcbf36ec1"} Oct 30 13:08:00.987017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.986859 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdc4a6ae6540d79f12d97ffe76c6ee79e6f88d5fe45055d16ab1802dcbf36ec1" Oct 30 13:08:00.987017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:00.986803 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7" Oct 30 13:08:01.040534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:01.040502 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 13:08:01.040746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:01.040558 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 13:08:01.313908 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:01.312809 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:02.241607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.241571 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:08:02.243519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.243496 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:08:02.244967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.244916 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:08:02.318483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.317109 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:08:02.318483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.318315 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:08:02.319257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.319231 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:08:02.714621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.714589 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf"] Oct 30 13:08:02.715221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.715207 5049 cpu_manager.go:414] "containerMap: removing container" podUID="80e20f8a-c8b2-4563-a0f9-25afdc0cea37" containerName="revoke-keys" Oct 30 13:08:02.715270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.715222 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e20f8a-c8b2-4563-a0f9-25afdc0cea37" containerName="revoke-keys" Oct 30 13:08:02.715270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.715239 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c0536a8a-196f-4607-bf42-84e47e979236" containerName="revoke-keys" Oct 30 13:08:02.715270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.715245 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0536a8a-196f-4607-bf42-84e47e979236" containerName="revoke-keys" Oct 30 13:08:02.715409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.715352 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0536a8a-196f-4607-bf42-84e47e979236" containerName="revoke-keys" Oct 30 13:08:02.715409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.715362 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e20f8a-c8b2-4563-a0f9-25afdc0cea37" containerName="revoke-keys" Oct 30 13:08:02.719970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.719945 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:02.738919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.738896 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:02.739283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.739235 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:08:02.743352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.742308 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf"] Oct 30 13:08:02.812309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.812270 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swj7z\" (UniqueName: \"kubernetes.io/projected/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-kube-api-access-swj7z\") pod \"maas-api-revoke-sub-f2382e43-zj2lf\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:02.812520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.812420 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2382e43-zj2lf\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:02.812579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.812532 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2382e43-zj2lf\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:02.913556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.913527 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2382e43-zj2lf\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:02.913701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.913639 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2382e43-zj2lf\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:02.913787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.913774 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swj7z\" (UniqueName: \"kubernetes.io/projected/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-kube-api-access-swj7z\") pod \"maas-api-revoke-sub-f2382e43-zj2lf\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:02.914449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.914421 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2382e43-zj2lf\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:02.915939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.915920 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2382e43-zj2lf\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:02.939598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:02.939569 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swj7z\" (UniqueName: \"kubernetes.io/projected/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-kube-api-access-swj7z\") pod \"maas-api-revoke-sub-f2382e43-zj2lf\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:03.033847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:03.033769 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:03.153850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:03.153817 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:03.344041 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:03.344005 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a56c78_f6e1_4315_b2a4_a0e6e5b437b7.slice/crio-25ce31c606b9dc4923923c1b70804b06e4fce4c97378824207333c69824eb7e3 WatchSource:0}: Error finding container 25ce31c606b9dc4923923c1b70804b06e4fce4c97378824207333c69824eb7e3: Status 404 returned error can't find the container with id 25ce31c606b9dc4923923c1b70804b06e4fce4c97378824207333c69824eb7e3 Oct 30 13:08:03.345573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:03.345540 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf"] Oct 30 13:08:04.014550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:04.014457 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" event={"ID":"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7","Type":"ContainerStarted","Data":"797d1abd0181d37b27d06e474634e68d4b0e062311872ed086996732c4d70bc5"} Oct 30 13:08:04.014550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:04.014520 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" event={"ID":"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7","Type":"ContainerStarted","Data":"25ce31c606b9dc4923923c1b70804b06e4fce4c97378824207333c69824eb7e3"} Oct 30 13:08:04.055809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:04.055756 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" podStartSLOduration=2.055742031 podStartE2EDuration="2.055742031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:04.055529782 +0000 UTC m=+990.870724206" watchObservedRunningTime="2026-10-30 13:08:04.055742031 +0000 UTC m=+990.870936453" Oct 30 13:08:05.023978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:05.023938 5049 generic.go:356] "Generic (PLEG): container finished" podID="f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7" containerID="797d1abd0181d37b27d06e474634e68d4b0e062311872ed086996732c4d70bc5" exitCode=0 Oct 30 13:08:05.024397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:05.024026 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" event={"ID":"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7","Type":"ContainerDied","Data":"797d1abd0181d37b27d06e474634e68d4b0e062311872ed086996732c4d70bc5"} Oct 30 13:08:05.100102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:05.100061 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:06.192696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.192668 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:06.357170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.357135 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-xks-ca\") pod \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " Oct 30 13:08:06.357409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.357388 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-openshift-ca\") pod \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " Oct 30 13:08:06.357478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.357463 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-kube-api-access-swj7z\" (UniqueName: \"kubernetes.io/projected/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-kube-api-access-swj7z\") pod \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\" (UID: \"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7\") " Oct 30 13:08:06.357767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.357744 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-openshift-ca" pod "f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7" (UID: "f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:06.357930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.357912 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:06.359445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.359415 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-kube-api-access-swj7z" pod "f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7" (UID: "f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7"). InnerVolumeSpecName "kube-api-access-swj7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:06.359564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.359442 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-xks-ca" pod "f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7" (UID: "f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:06.459388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.459348 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swj7z\" (UniqueName: \"kubernetes.io/projected/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-kube-api-access-swj7z\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:06.459388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.459378 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:06.906976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:06.906933 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:07.041349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.041289 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" Oct 30 13:08:07.041518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.041312 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf" event={"ID":"f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7","Type":"ContainerDied","Data":"25ce31c606b9dc4923923c1b70804b06e4fce4c97378824207333c69824eb7e3"} Oct 30 13:08:07.041518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.041397 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25ce31c606b9dc4923923c1b70804b06e4fce4c97378824207333c69824eb7e3" Oct 30 13:08:07.191435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.191356 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9"] Oct 30 13:08:07.192047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.192030 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7" containerName="revoke-keys" Oct 30 13:08:07.192096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.192049 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7" containerName="revoke-keys" Oct 30 13:08:07.192167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.192156 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7" containerName="revoke-keys" Oct 30 13:08:07.196103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.196079 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.213217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.213192 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:07.213517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.213497 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:08:07.219911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.219466 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9"] Oct 30 13:08:07.269211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.269175 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-012bfe7c-zcsm9\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.269395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.269226 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-012bfe7c-zcsm9\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.269469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.269450 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsp97\" (UniqueName: \"kubernetes.io/projected/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-kube-api-access-dsp97\") pod \"maas-api-revoke-sub-012bfe7c-zcsm9\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.370862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.370823 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-012bfe7c-zcsm9\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.371049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.370922 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-012bfe7c-zcsm9\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.371377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.371310 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsp97\" (UniqueName: \"kubernetes.io/projected/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-kube-api-access-dsp97\") pod \"maas-api-revoke-sub-012bfe7c-zcsm9\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.371822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.371802 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-012bfe7c-zcsm9\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.373228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.373209 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-012bfe7c-zcsm9\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.395315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.395286 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsp97\" (UniqueName: \"kubernetes.io/projected/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-kube-api-access-dsp97\") pod \"maas-api-revoke-sub-012bfe7c-zcsm9\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.510705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.510615 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:07.823272 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:07.823235 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbdcd3d_b29e_4138_bbea_b723c3377b9e.slice/crio-762872ba90f312ada1459dc098f1937631c4f40b3a22dd381c631ac7f521d4b9 WatchSource:0}: Error finding container 762872ba90f312ada1459dc098f1937631c4f40b3a22dd381c631ac7f521d4b9: Status 404 returned error can't find the container with id 762872ba90f312ada1459dc098f1937631c4f40b3a22dd381c631ac7f521d4b9 Oct 30 13:08:07.824098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:07.824069 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9"] Oct 30 13:08:08.051577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.051533 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" event={"ID":"1fbdcd3d-b29e-4138-bbea-b723c3377b9e","Type":"ContainerStarted","Data":"762872ba90f312ada1459dc098f1937631c4f40b3a22dd381c631ac7f521d4b9"} Oct 30 13:08:08.703296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.703260 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9"] Oct 30 13:08:08.707664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.707637 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.747010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.746981 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9"] Oct 30 13:08:08.790216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.790186 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.790388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.790246 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb76201-354c-4b8c-8f6b-2483135657c6-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.790460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.790387 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.790460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.790425 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88987\" (UniqueName: \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-kube-api-access-88987\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.790581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.790563 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.790640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.790610 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.790689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.790672 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.790750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.790738 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.790785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.790767 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.861637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.860630 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:08.891796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.891712 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.891796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.891756 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.891880 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.891982 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb76201-354c-4b8c-8f6b-2483135657c6-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.892029 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.892056 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88987\" (UniqueName: \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-kube-api-access-88987\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.892096 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.892172 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.892187 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.892206 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.892240 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.892497 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.892646 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.892726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.892686 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb76201-354c-4b8c-8f6b-2483135657c6-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.894400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.894381 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.894806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.894784 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.915454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.915429 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88987\" (UniqueName: \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-kube-api-access-88987\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:08.915572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:08.915465 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:09.062055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:09.062020 5049 generic.go:356] "Generic (PLEG): container finished" podID="1fbdcd3d-b29e-4138-bbea-b723c3377b9e" containerID="0c6c9dac92a0b9ece5ded130ae876bd63f3a988486c8156998f5b55fabc0e106" exitCode=0 Oct 30 13:08:09.062242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:09.062109 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" event={"ID":"1fbdcd3d-b29e-4138-bbea-b723c3377b9e","Type":"ContainerDied","Data":"0c6c9dac92a0b9ece5ded130ae876bd63f3a988486c8156998f5b55fabc0e106"} Oct 30 13:08:09.092956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:09.092927 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:09.433196 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:09.433168 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb76201_354c_4b8c_8f6b_2483135657c6.slice/crio-616d70b2db36b459607a5ab0c598427cd7a5a75904e833c6faf098b48cb258ee WatchSource:0}: Error finding container 616d70b2db36b459607a5ab0c598427cd7a5a75904e833c6faf098b48cb258ee: Status 404 returned error can't find the container with id 616d70b2db36b459607a5ab0c598427cd7a5a75904e833c6faf098b48cb258ee Oct 30 13:08:09.433568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:09.433539 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9"] Oct 30 13:08:09.435939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:09.435908 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:09.436041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:09.435985 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:09.436041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:09.436028 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:10.099782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.094116 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" event={"ID":"cbb76201-354c-4b8c-8f6b-2483135657c6","Type":"ContainerStarted","Data":"591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98"} Oct 30 13:08:10.099782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.094188 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" event={"ID":"cbb76201-354c-4b8c-8f6b-2483135657c6","Type":"ContainerStarted","Data":"616d70b2db36b459607a5ab0c598427cd7a5a75904e833c6faf098b48cb258ee"} Oct 30 13:08:10.183344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.179202 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podStartSLOduration=2.179187748 podStartE2EDuration="2.179187748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:10.164203019 +0000 UTC m=+996.979397439" watchObservedRunningTime="2026-10-30 13:08:10.179187748 +0000 UTC m=+996.994382170" Oct 30 13:08:10.794346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.793659 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:10.820345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.819615 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:10.894397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.893382 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-openshift-ca\") pod \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " Oct 30 13:08:10.894397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.893484 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-kube-api-access-dsp97\" (UniqueName: \"kubernetes.io/projected/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-kube-api-access-dsp97\") pod \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " Oct 30 13:08:10.894397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.893557 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-xks-ca\") pod \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\" (UID: \"1fbdcd3d-b29e-4138-bbea-b723c3377b9e\") " Oct 30 13:08:10.917343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.894676 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-openshift-ca" pod "1fbdcd3d-b29e-4138-bbea-b723c3377b9e" (UID: "1fbdcd3d-b29e-4138-bbea-b723c3377b9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:10.917343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.915504 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-xks-ca" pod "1fbdcd3d-b29e-4138-bbea-b723c3377b9e" (UID: "1fbdcd3d-b29e-4138-bbea-b723c3377b9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:10.917343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:10.916093 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-kube-api-access-dsp97" pod "1fbdcd3d-b29e-4138-bbea-b723c3377b9e" (UID: "1fbdcd3d-b29e-4138-bbea-b723c3377b9e"). InnerVolumeSpecName "kube-api-access-dsp97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:11.006346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.005499 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:11.006346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.005534 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:11.006346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.005548 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsp97\" (UniqueName: \"kubernetes.io/projected/1fbdcd3d-b29e-4138-bbea-b723c3377b9e-kube-api-access-dsp97\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:11.145401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.134132 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:11.174343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.173341 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:08:11.174343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.173413 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 13:08:11.243349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.240834 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" Oct 30 13:08:11.281346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.261515 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9" event={"ID":"1fbdcd3d-b29e-4138-bbea-b723c3377b9e","Type":"ContainerDied","Data":"762872ba90f312ada1459dc098f1937631c4f40b3a22dd381c631ac7f521d4b9"} Oct 30 13:08:11.281346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.261584 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="762872ba90f312ada1459dc098f1937631c4f40b3a22dd381c631ac7f521d4b9" Oct 30 13:08:11.829646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.828803 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:08:11.841569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:11.841159 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:08:12.043357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.041529 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:12.099407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.095750 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:08:12.099407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.095824 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 13:08:12.324190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.324151 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:12.348358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.343477 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:12.355776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.353979 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 13:08:12.428280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.426921 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:12.458563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.458429 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:12.458563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.458497 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:12.501347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.500672 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:08:12.830526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.830465 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:08:12.831731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.831691 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:08:12.852041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.852000 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:12.852713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:12.852688 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:08:13.055348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.051931 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:08:13.055348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.052802 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:08:13.119409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.102770 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:08:13.119409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.102832 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 13:08:13.361351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.360470 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:13.386429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.362625 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:13.386429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.363091 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:13.458355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.449698 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:13.654409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.629129 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:13.713429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.686123 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 30 13:08:13.802951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.802911 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:08:13.838609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.838545 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:13.865260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.863991 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:08:13.885380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:13.876085 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:14.083283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.071754 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 13:08:14.083283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.075531 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:08:14.141849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.141476 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:08:14.141849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.141535 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 13:08:14.386396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.361509 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:14.417398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.368287 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:08:14.418080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.400664 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:14.473055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.473010 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:08:14.473466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.473437 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:08:14.487822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.487795 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:08:14.488133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.488103 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:08:14.489212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.474352 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:14.596346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.580890 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:08:14.700479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.700366 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:14.788903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.781871 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:08:14.792336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.792296 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj"] Oct 30 13:08:14.793450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.793429 5049 cpu_manager.go:414] "containerMap: removing container" podUID="1fbdcd3d-b29e-4138-bbea-b723c3377b9e" containerName="revoke-keys" Oct 30 13:08:14.793586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.793570 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fbdcd3d-b29e-4138-bbea-b723c3377b9e" containerName="revoke-keys" Oct 30 13:08:14.793920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.793902 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fbdcd3d-b29e-4138-bbea-b723c3377b9e" containerName="revoke-keys" Oct 30 13:08:14.798648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.798628 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:14.815418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.815271 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"distinct-w2-cb58b8-kserve-self-signed-certs\"" Oct 30 13:08:14.837518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.837437 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj"] Oct 30 13:08:14.841655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.840400 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:14.887406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.879273 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:08:14.915354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.913016 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-home\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:14.915354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.913129 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-model-cache\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:14.915354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.913229 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22fa542-32e1-4074-9d28-cadb02442c7b-tls-certs\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:14.915354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.913276 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-tmp-dir\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:14.915354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.913302 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtml\" (UniqueName: \"kubernetes.io/projected/a22fa542-32e1-4074-9d28-cadb02442c7b-kube-api-access-hmtml\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:14.915354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:14.913427 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-dshm\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.031940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.031843 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22fa542-32e1-4074-9d28-cadb02442c7b-tls-certs\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.033149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.033121 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-tmp-dir\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.033361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.033344 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtml\" (UniqueName: \"kubernetes.io/projected/a22fa542-32e1-4074-9d28-cadb02442c7b-kube-api-access-hmtml\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.033477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.033463 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-dshm\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.033925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.033660 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-home\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.034179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.034157 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-model-cache\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.070584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.035759 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-tmp-dir\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.070857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.056793 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-model-cache\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.070997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.032681 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:08:15.086349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.059074 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-home\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.086566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.074590 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:15.087258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.082755 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:15.090200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.090145 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:08:15.097549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.097470 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-dshm\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.101742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.099230 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:08:15.103418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.102588 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:08:15.103418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.102637 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 13:08:15.103418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.103221 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22fa542-32e1-4074-9d28-cadb02442c7b-tls-certs\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.111350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.110718 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtml\" (UniqueName: \"kubernetes.io/projected/a22fa542-32e1-4074-9d28-cadb02442c7b-kube-api-access-hmtml\") pod \"distinct-w2-cb58b8-kserve-7696cf655c-7xzvj\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.159345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.153572 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:08:15.178080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.178044 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:08:15.263563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.263522 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:08:15.268024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.267990 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:15.691563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.691495 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj"] Oct 30 13:08:15.697732 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:15.692148 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda22fa542_32e1_4074_9d28_cadb02442c7b.slice/crio-0764258fa4922cf4eb330e749ce41bc8d917025ac3b0f1a6fe4a713b601cda59 WatchSource:0}: Error finding container 0764258fa4922cf4eb330e749ce41bc8d917025ac3b0f1a6fe4a713b601cda59: Status 404 returned error can't find the container with id 0764258fa4922cf4eb330e749ce41bc8d917025ac3b0f1a6fe4a713b601cda59 Oct 30 13:08:15.934129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:15.934089 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:16.004613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:16.004524 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" event={"ID":"a22fa542-32e1-4074-9d28-cadb02442c7b","Type":"ContainerStarted","Data":"33b4c95539c43d800e7e55634883ca697cfab7ffb5b823bd2882d7d8ec60a7f7"} Oct 30 13:08:16.004613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:16.004577 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" event={"ID":"a22fa542-32e1-4074-9d28-cadb02442c7b","Type":"ContainerStarted","Data":"0764258fa4922cf4eb330e749ce41bc8d917025ac3b0f1a6fe4a713b601cda59"} Oct 30 13:08:16.054193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:16.054137 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podStartSLOduration=2.054122302 podStartE2EDuration="2.054122302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:16.052431084 +0000 UTC m=+1002.867625507" watchObservedRunningTime="2026-10-30 13:08:16.054122302 +0000 UTC m=+1002.869316731" Oct 30 13:08:16.094175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:16.094141 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:08:16.094398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:16.094193 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 13:08:16.268634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:16.268555 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:17.093953 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:17.093922 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:08:17.094356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:17.093972 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 13:08:17.283793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:17.283735 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:17.999712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:17.999662 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:18.094250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:18.094216 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:08:18.094661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:18.094272 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 13:08:19.792168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:19.792122 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:20.134518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.134471 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:20.134706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.134567 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:20.157979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.157947 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv"] Oct 30 13:08:20.162583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.162564 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.186673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.186645 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv"] Oct 30 13:08:20.192094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.192066 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:20.192754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.192731 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:08:20.240909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.240870 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffn2z\" (UniqueName: \"kubernetes.io/projected/bfa69f85-0f28-40c9-b531-6ce96b3c074f-kube-api-access-ffn2z\") pod \"maas-api-revoke-sub-fee525a9-ksbdv\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.241166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.241148 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fee525a9-ksbdv\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.241280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.241267 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fee525a9-ksbdv\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.322612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.322579 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:20.323016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.322987 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:20.324111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.324088 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:08:20.342315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.342290 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fee525a9-ksbdv\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.342436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.342335 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fee525a9-ksbdv\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.342655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.342630 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffn2z\" (UniqueName: \"kubernetes.io/projected/bfa69f85-0f28-40c9-b531-6ce96b3c074f-kube-api-access-ffn2z\") pod \"maas-api-revoke-sub-fee525a9-ksbdv\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.343099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.343066 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fee525a9-ksbdv\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.344950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.344917 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fee525a9-ksbdv\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.367135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.367105 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffn2z\" (UniqueName: \"kubernetes.io/projected/bfa69f85-0f28-40c9-b531-6ce96b3c074f-kube-api-access-ffn2z\") pod \"maas-api-revoke-sub-fee525a9-ksbdv\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.478627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.478530 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:20.490389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.490360 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6"] Oct 30 13:08:20.495134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.495111 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:20.527221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.527192 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6"] Oct 30 13:08:20.570181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.570144 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a70454-h7vv6\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:20.570366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.570282 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a70454-h7vv6\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:20.570442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.570398 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6br8\" (UniqueName: \"kubernetes.io/projected/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-kube-api-access-g6br8\") pod \"maas-api-revoke-sub-83a70454-h7vv6\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:20.672252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.672213 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a70454-h7vv6\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:20.672413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.672370 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a70454-h7vv6\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:20.672490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.672472 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6br8\" (UniqueName: \"kubernetes.io/projected/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-kube-api-access-g6br8\") pod \"maas-api-revoke-sub-83a70454-h7vv6\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:20.673030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.673004 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a70454-h7vv6\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:20.674788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.674769 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a70454-h7vv6\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:20.700736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.700714 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6br8\" (UniqueName: \"kubernetes.io/projected/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-kube-api-access-g6br8\") pod \"maas-api-revoke-sub-83a70454-h7vv6\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:20.828532 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:20.828489 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa69f85_0f28_40c9_b531_6ce96b3c074f.slice/crio-b5463cc499e035aa1e60c8fae725ceb5d7e5cd3cef8c0afdb42fef7f5bd06c43 WatchSource:0}: Error finding container b5463cc499e035aa1e60c8fae725ceb5d7e5cd3cef8c0afdb42fef7f5bd06c43: Status 404 returned error can't find the container with id b5463cc499e035aa1e60c8fae725ceb5d7e5cd3cef8c0afdb42fef7f5bd06c43 Oct 30 13:08:20.830050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.829929 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv"] Oct 30 13:08:20.834993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:20.834971 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:21.062218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:21.062183 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" event={"ID":"bfa69f85-0f28-40c9-b531-6ce96b3c074f","Type":"ContainerStarted","Data":"b5463cc499e035aa1e60c8fae725ceb5d7e5cd3cef8c0afdb42fef7f5bd06c43"} Oct 30 13:08:21.180495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:21.180465 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6"] Oct 30 13:08:21.180700 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:21.180679 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cdaa62_2ef9_42ac_9469_8c1df36f8d5d.slice/crio-e10201ab2311a161dc6e33cf3f9ee72a495624ae9b8421dff113371bf398ea1b WatchSource:0}: Error finding container e10201ab2311a161dc6e33cf3f9ee72a495624ae9b8421dff113371bf398ea1b: Status 404 returned error can't find the container with id e10201ab2311a161dc6e33cf3f9ee72a495624ae9b8421dff113371bf398ea1b Oct 30 13:08:21.900372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:21.899950 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:22.071511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:22.071409 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" event={"ID":"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d","Type":"ContainerStarted","Data":"4e56b0e13d07f8394b6f1309ece3661a6def992d417e4095de6b05cba416129f"} Oct 30 13:08:22.071511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:22.071469 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" event={"ID":"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d","Type":"ContainerStarted","Data":"e10201ab2311a161dc6e33cf3f9ee72a495624ae9b8421dff113371bf398ea1b"} Oct 30 13:08:22.072947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:22.072919 5049 generic.go:356] "Generic (PLEG): container finished" podID="bfa69f85-0f28-40c9-b531-6ce96b3c074f" containerID="7de24a50ecfa00cc1b25f8a4d41cd3e875a1694c5ccda63b3c6ac3e4182228b0" exitCode=0 Oct 30 13:08:22.073064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:22.072985 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" event={"ID":"bfa69f85-0f28-40c9-b531-6ce96b3c074f","Type":"ContainerDied","Data":"7de24a50ecfa00cc1b25f8a4d41cd3e875a1694c5ccda63b3c6ac3e4182228b0"} Oct 30 13:08:22.115473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:22.115420 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" podStartSLOduration=2.115405427 podStartE2EDuration="2.115405427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:22.113519148 +0000 UTC m=+1008.928713572" watchObservedRunningTime="2026-10-30 13:08:22.115405427 +0000 UTC m=+1008.930599850" Oct 30 13:08:23.083242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.083209 5049 generic.go:356] "Generic (PLEG): container finished" podID="b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d" containerID="4e56b0e13d07f8394b6f1309ece3661a6def992d417e4095de6b05cba416129f" exitCode=0 Oct 30 13:08:23.083795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.083281 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" event={"ID":"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d","Type":"ContainerDied","Data":"4e56b0e13d07f8394b6f1309ece3661a6def992d417e4095de6b05cba416129f"} Oct 30 13:08:23.122735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.122702 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8"] Oct 30 13:08:23.127603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.127575 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.182731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.182643 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8"] Oct 30 13:08:23.215601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.214922 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.215601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.214970 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.215601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.215060 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.215601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.215084 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a985f33-13d8-4e26-aa45-0709786d0fb7-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.215601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.215131 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.215601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.215158 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.215601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.215207 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.215601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.215237 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxp7\" (UniqueName: \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-kube-api-access-5rxp7\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.215601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.215269 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.302601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.302576 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:23.316536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.316505 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxp7\" (UniqueName: \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-kube-api-access-5rxp7\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.316696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.316545 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.316696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.316661 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.316696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.316678 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.316863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.316737 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.316863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.316752 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a985f33-13d8-4e26-aa45-0709786d0fb7-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.316863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.316783 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.316863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.316804 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.316863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.316835 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.317113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.317039 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.317168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.317126 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.317455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.317429 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.318292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.318263 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a985f33-13d8-4e26-aa45-0709786d0fb7-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.318428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.318409 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.319165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.319138 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.319619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.319595 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.351376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.351268 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.352759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.352735 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxp7\" (UniqueName: \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-kube-api-access-5rxp7\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.418255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.418216 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa69f85-0f28-40c9-b531-6ce96b3c074f-kube-api-access-ffn2z\" (UniqueName: \"kubernetes.io/projected/bfa69f85-0f28-40c9-b531-6ce96b3c074f-kube-api-access-ffn2z\") pod \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " Oct 30 13:08:23.418481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.418284 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-xks-ca\") pod \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " Oct 30 13:08:23.418481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.418384 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-openshift-ca\") pod \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\" (UID: \"bfa69f85-0f28-40c9-b531-6ce96b3c074f\") " Oct 30 13:08:23.419078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.418762 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-openshift-ca" pod "bfa69f85-0f28-40c9-b531-6ce96b3c074f" (UID: "bfa69f85-0f28-40c9-b531-6ce96b3c074f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:23.419282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.419261 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:23.420633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.420607 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa69f85-0f28-40c9-b531-6ce96b3c074f-kube-api-access-ffn2z" pod "bfa69f85-0f28-40c9-b531-6ce96b3c074f" (UID: "bfa69f85-0f28-40c9-b531-6ce96b3c074f"). InnerVolumeSpecName "kube-api-access-ffn2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:23.420718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.420613 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-xks-ca" pod "bfa69f85-0f28-40c9-b531-6ce96b3c074f" (UID: "bfa69f85-0f28-40c9-b531-6ce96b3c074f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:23.459576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.459538 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:23.520911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.520883 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffn2z\" (UniqueName: \"kubernetes.io/projected/bfa69f85-0f28-40c9-b531-6ce96b3c074f-kube-api-access-ffn2z\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:23.521022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.520912 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa69f85-0f28-40c9-b531-6ce96b3c074f-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:23.752518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.752412 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:23.797955 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:23.797925 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a985f33_13d8_4e26_aa45_0709786d0fb7.slice/crio-e26eddc2829c4d5505e5b198a56ca251aecffe6050825343b467544736f1e274 WatchSource:0}: Error finding container e26eddc2829c4d5505e5b198a56ca251aecffe6050825343b467544736f1e274: Status 404 returned error can't find the container with id e26eddc2829c4d5505e5b198a56ca251aecffe6050825343b467544736f1e274 Oct 30 13:08:23.798359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.798336 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8"] Oct 30 13:08:23.800289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.800258 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:23.800407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.800345 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:23.800407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.800380 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:23.978890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:23.978851 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:24.094176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.094137 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" event={"ID":"8a985f33-13d8-4e26-aa45-0709786d0fb7","Type":"ContainerStarted","Data":"ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798"} Oct 30 13:08:24.094687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.094204 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" event={"ID":"8a985f33-13d8-4e26-aa45-0709786d0fb7","Type":"ContainerStarted","Data":"e26eddc2829c4d5505e5b198a56ca251aecffe6050825343b467544736f1e274"} Oct 30 13:08:24.095790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.095761 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" event={"ID":"bfa69f85-0f28-40c9-b531-6ce96b3c074f","Type":"ContainerDied","Data":"b5463cc499e035aa1e60c8fae725ceb5d7e5cd3cef8c0afdb42fef7f5bd06c43"} Oct 30 13:08:24.095992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.095805 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5463cc499e035aa1e60c8fae725ceb5d7e5cd3cef8c0afdb42fef7f5bd06c43" Oct 30 13:08:24.095992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.095776 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv" Oct 30 13:08:24.155700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.155639 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podStartSLOduration=1.155620134 podStartE2EDuration="1.155620134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:24.151639842 +0000 UTC m=+1010.966834266" watchObservedRunningTime="2026-10-30 13:08:24.155620134 +0000 UTC m=+1010.970814557" Oct 30 13:08:24.271751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.271725 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:24.333432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.333398 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-xks-ca\") pod \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " Oct 30 13:08:24.333595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.333504 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-openshift-ca\") pod \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " Oct 30 13:08:24.333673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.333650 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-kube-api-access-g6br8\" (UniqueName: \"kubernetes.io/projected/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-kube-api-access-g6br8\") pod \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\" (UID: \"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d\") " Oct 30 13:08:24.334187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.334149 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-openshift-ca" pod "b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d" (UID: "b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:24.336004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.335979 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-xks-ca" pod "b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d" (UID: "b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:24.336456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.336435 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-kube-api-access-g6br8" pod "b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d" (UID: "b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d"). InnerVolumeSpecName "kube-api-access-g6br8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:24.436129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.436090 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6br8\" (UniqueName: \"kubernetes.io/projected/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-kube-api-access-g6br8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:24.436129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.436132 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:24.436406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.436149 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:24.460091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.460059 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:24.461579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.461555 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:08:24.461679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:24.461605 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 13:08:25.122345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:25.106535 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" Oct 30 13:08:25.122345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:25.110469 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6" event={"ID":"b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d","Type":"ContainerDied","Data":"e10201ab2311a161dc6e33cf3f9ee72a495624ae9b8421dff113371bf398ea1b"} Oct 30 13:08:25.131526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:25.129428 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e10201ab2311a161dc6e33cf3f9ee72a495624ae9b8421dff113371bf398ea1b" Oct 30 13:08:25.131526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:25.131409 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:25.268786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:25.268699 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:25.281997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:25.281970 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:25.460668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:25.460636 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:08:25.460814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:25.460686 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 13:08:25.971697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:25.971660 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:26.255458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:26.254375 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:26.460970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:26.460234 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:08:26.460970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:26.460436 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 13:08:26.788137 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:08:26.788097 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:08:27.463340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:27.462867 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:08:27.463340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:27.462946 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 13:08:28.462669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:28.462633 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:08:28.462978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:28.462949 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 13:08:28.528580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:28.528536 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:28.838377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:28.838312 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:28.877615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:28.876438 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:28.973769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:28.973433 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:29.040905 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:29.040421 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:29.115961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:29.115860 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:29.163836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:29.163795 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:29.164600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:29.164572 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:29.189205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:29.189143 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:29.241390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:29.221408 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:29.913375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:29.835500 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:08:29.913375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:29.835585 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 13:08:30.168423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.147965 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:08:30.210398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.197631 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:08:30.257465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.257422 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:30.321137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.259498 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:30.322080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.318585 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:08:30.322624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.319253 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:30.353346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.316442 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:30.463602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.463470 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:08:30.482606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.482557 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:08:30.482887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.482855 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 13:08:30.492868 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:08:30.492835 5049 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.147s" Oct 30 13:08:30.519034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.519009 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:08:30.548508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.548476 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:08:30.548702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.548556 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:08:30.548702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.548589 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:08:30.548702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.548618 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:08:30.548702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.548652 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:08:30.548702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.548682 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:30.548962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.548710 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:08:30.548962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.548739 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:08:30.568917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.568891 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:30.603251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.602822 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:08:30.657927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.650429 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:30.797461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.781020 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:08:30.797461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.781744 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:08:30.800058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.800020 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:08:30.910032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:30.909905 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:31.460009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.459966 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:08:31.460624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.460591 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 13:08:31.556801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.556755 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7"] Oct 30 13:08:31.557789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.557756 5049 cpu_manager.go:414] "containerMap: removing container" podUID="bfa69f85-0f28-40c9-b531-6ce96b3c074f" containerName="revoke-keys" Oct 30 13:08:31.557789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.557783 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa69f85-0f28-40c9-b531-6ce96b3c074f" containerName="revoke-keys" Oct 30 13:08:31.557993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.557804 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d" containerName="revoke-keys" Oct 30 13:08:31.557993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.557813 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d" containerName="revoke-keys" Oct 30 13:08:31.557993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.557969 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa69f85-0f28-40c9-b531-6ce96b3c074f" containerName="revoke-keys" Oct 30 13:08:31.557993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.557982 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d" containerName="revoke-keys" Oct 30 13:08:31.568090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.568059 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.588438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.588406 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7"] Oct 30 13:08:31.592541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.591307 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 13:08:31.592703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.591365 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 13:08:31.595496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.593030 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 13:08:31.595496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.593192 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-glws9\"" Oct 30 13:08:31.691022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.690986 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.691235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.691048 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.691309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.691239 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f8396f-7488-447c-97f9-2073e0d6199c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.691309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.691275 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.691446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.691423 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bp9r\" (UniqueName: \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-kube-api-access-6bp9r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.691504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.691477 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.691587 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.691575 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.792724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.792631 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f8396f-7488-447c-97f9-2073e0d6199c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.792724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.792674 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.792950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.792748 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bp9r\" (UniqueName: \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-kube-api-access-6bp9r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.792950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.792787 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.793221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.793198 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.793359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.793342 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.793432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.793418 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.794400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.794369 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.795796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.795681 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.795796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.795713 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f8396f-7488-447c-97f9-2073e0d6199c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.795975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.795853 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.795975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.795970 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.819954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.819922 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.820802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.820774 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bp9r\" (UniqueName: \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-kube-api-access-6bp9r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82sf7\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:31.925407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:31.925372 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:32.271468 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:32.271437 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f8396f_7488_447c_97f9_2073e0d6199c.slice/crio-e9aa6803999dc0ecb9cadd13a5e78e75b3869338f3ce63185b72223afe06e1db WatchSource:0}: Error finding container e9aa6803999dc0ecb9cadd13a5e78e75b3869338f3ce63185b72223afe06e1db: Status 404 returned error can't find the container with id e9aa6803999dc0ecb9cadd13a5e78e75b3869338f3ce63185b72223afe06e1db Oct 30 13:08:32.273886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:32.273856 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7"] Oct 30 13:08:32.274292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:32.274253 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:32.274383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:32.274363 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:32.426854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:32.426818 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" event={"ID":"a6f8396f-7488-447c-97f9-2073e0d6199c","Type":"ContainerStarted","Data":"70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459"} Oct 30 13:08:32.427029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:32.426873 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" event={"ID":"a6f8396f-7488-447c-97f9-2073e0d6199c","Type":"ContainerStarted","Data":"e9aa6803999dc0ecb9cadd13a5e78e75b3869338f3ce63185b72223afe06e1db"} Oct 30 13:08:32.427029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:32.426908 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:32.478525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:32.478476 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" podStartSLOduration=1.4784607379999999 podStartE2EDuration="1.478460738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:32.473456042 +0000 UTC m=+1019.288650466" watchObservedRunningTime="2026-10-30 13:08:32.478460738 +0000 UTC m=+1019.293655161" Oct 30 13:08:32.821541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:32.821505 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:33.448371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:33.445842 5049 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-82sf7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:08:33.448371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:33.445894 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" podUID="a6f8396f-7488-447c-97f9-2073e0d6199c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:08:33.524345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:33.504410 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:33.524345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:33.504495 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:34.154379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.152813 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm"] Oct 30 13:08:34.159568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.159531 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.172154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.172123 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5216becf-215f-4045-a425-3bca932690f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-905c6e55-p4pmm\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.172315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.172166 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5216becf-215f-4045-a425-3bca932690f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-905c6e55-p4pmm\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.172421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.172345 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhqxz\" (UniqueName: \"kubernetes.io/projected/5216becf-215f-4045-a425-3bca932690f1-kube-api-access-qhqxz\") pod \"maas-api-revoke-sub-905c6e55-p4pmm\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.181463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.181435 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm"] Oct 30 13:08:34.186608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.186586 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:08:34.186608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.186586 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:34.210708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.210679 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:34.211518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.211496 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:34.212598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.212571 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:08:34.273688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.273650 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5216becf-215f-4045-a425-3bca932690f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-905c6e55-p4pmm\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.273688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.273686 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5216becf-215f-4045-a425-3bca932690f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-905c6e55-p4pmm\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.273926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.273750 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhqxz\" (UniqueName: \"kubernetes.io/projected/5216becf-215f-4045-a425-3bca932690f1-kube-api-access-qhqxz\") pod \"maas-api-revoke-sub-905c6e55-p4pmm\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.274441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.274416 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5216becf-215f-4045-a425-3bca932690f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-905c6e55-p4pmm\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.276094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.276071 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5216becf-215f-4045-a425-3bca932690f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-905c6e55-p4pmm\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.303652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.303626 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhqxz\" (UniqueName: \"kubernetes.io/projected/5216becf-215f-4045-a425-3bca932690f1-kube-api-access-qhqxz\") pod \"maas-api-revoke-sub-905c6e55-p4pmm\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.474180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.474098 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:34.799925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.799900 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm"] Oct 30 13:08:34.800957 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:34.800922 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5216becf_215f_4045_a425_3bca932690f1.slice/crio-51ca860bda7dd3b88d6ac0fd173ba7d091c5b38f455f438f81bdf6d3d39e7e6c WatchSource:0}: Error finding container 51ca860bda7dd3b88d6ac0fd173ba7d091c5b38f455f438f81bdf6d3d39e7e6c: Status 404 returned error can't find the container with id 51ca860bda7dd3b88d6ac0fd173ba7d091c5b38f455f438f81bdf6d3d39e7e6c Oct 30 13:08:34.998163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:34.998120 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:35.458910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:35.458819 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" event={"ID":"5216becf-215f-4045-a425-3bca932690f1","Type":"ContainerStarted","Data":"25967713bca2ebb1fef62c94215b1c816382ccc6a35dae4165eee115300c111e"} Oct 30 13:08:35.458910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:35.458885 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" event={"ID":"5216becf-215f-4045-a425-3bca932690f1","Type":"ContainerStarted","Data":"51ca860bda7dd3b88d6ac0fd173ba7d091c5b38f455f438f81bdf6d3d39e7e6c"} Oct 30 13:08:35.536449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:35.536384 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" podStartSLOduration=1.536362628 podStartE2EDuration="1.536362628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:35.530778536 +0000 UTC m=+1022.345972959" watchObservedRunningTime="2026-10-30 13:08:35.536362628 +0000 UTC m=+1022.351557052" Oct 30 13:08:36.445778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:36.445748 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:08:36.468732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:36.468700 5049 generic.go:356] "Generic (PLEG): container finished" podID="5216becf-215f-4045-a425-3bca932690f1" containerID="25967713bca2ebb1fef62c94215b1c816382ccc6a35dae4165eee115300c111e" exitCode=0 Oct 30 13:08:36.468893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:36.468754 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" event={"ID":"5216becf-215f-4045-a425-3bca932690f1","Type":"ContainerDied","Data":"25967713bca2ebb1fef62c94215b1c816382ccc6a35dae4165eee115300c111e"} Oct 30 13:08:36.986253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:36.986218 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj"] Oct 30 13:08:36.990595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:36.990577 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.014214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.014179 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj"] Oct 30 13:08:37.116885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.116851 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q8s6\" (UniqueName: \"kubernetes.io/projected/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-kube-api-access-8q8s6\") pod \"maas-api-revoke-sub-027f93be-mjxsj\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.117051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.116987 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-027f93be-mjxsj\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.117051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.117013 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-027f93be-mjxsj\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.182465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.182430 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:37.218452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.218411 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q8s6\" (UniqueName: \"kubernetes.io/projected/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-kube-api-access-8q8s6\") pod \"maas-api-revoke-sub-027f93be-mjxsj\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.218637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.218567 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-027f93be-mjxsj\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.218637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.218598 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-027f93be-mjxsj\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.219395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.219366 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-027f93be-mjxsj\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.221308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.221278 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-027f93be-mjxsj\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.242380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.242296 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q8s6\" (UniqueName: \"kubernetes.io/projected/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-kube-api-access-8q8s6\") pod \"maas-api-revoke-sub-027f93be-mjxsj\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.304902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.304873 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:37.619703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.617378 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj"] Oct 30 13:08:37.621159 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:37.621135 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3dfa6d_8795_44c8_8b2e_1ccfc63021ef.slice/crio-acfffc9b20b91eba6445e857f65c01edda490d13dac32a7831d89f890af24f34 WatchSource:0}: Error finding container acfffc9b20b91eba6445e857f65c01edda490d13dac32a7831d89f890af24f34: Status 404 returned error can't find the container with id acfffc9b20b91eba6445e857f65c01edda490d13dac32a7831d89f890af24f34 Oct 30 13:08:37.643697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.643671 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:37.725252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.725209 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5216becf-215f-4045-a425-3bca932690f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5216becf-215f-4045-a425-3bca932690f1-maas-api-openshift-ca\") pod \"5216becf-215f-4045-a425-3bca932690f1\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " Oct 30 13:08:37.725466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.725378 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5216becf-215f-4045-a425-3bca932690f1-kube-api-access-qhqxz\" (UniqueName: \"kubernetes.io/projected/5216becf-215f-4045-a425-3bca932690f1-kube-api-access-qhqxz\") pod \"5216becf-215f-4045-a425-3bca932690f1\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " Oct 30 13:08:37.725466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.725445 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5216becf-215f-4045-a425-3bca932690f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5216becf-215f-4045-a425-3bca932690f1-maas-api-xks-ca\") pod \"5216becf-215f-4045-a425-3bca932690f1\" (UID: \"5216becf-215f-4045-a425-3bca932690f1\") " Oct 30 13:08:37.725686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.725655 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5216becf-215f-4045-a425-3bca932690f1-maas-api-openshift-ca" pod "5216becf-215f-4045-a425-3bca932690f1" (UID: "5216becf-215f-4045-a425-3bca932690f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:37.726068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.726042 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5216becf-215f-4045-a425-3bca932690f1-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:37.728121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.728088 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5216becf-215f-4045-a425-3bca932690f1-maas-api-xks-ca" pod "5216becf-215f-4045-a425-3bca932690f1" (UID: "5216becf-215f-4045-a425-3bca932690f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:37.728234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.728117 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5216becf-215f-4045-a425-3bca932690f1-kube-api-access-qhqxz" pod "5216becf-215f-4045-a425-3bca932690f1" (UID: "5216becf-215f-4045-a425-3bca932690f1"). InnerVolumeSpecName "kube-api-access-qhqxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:37.794157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.793953 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s"] Oct 30 13:08:37.795750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.794886 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5216becf-215f-4045-a425-3bca932690f1" containerName="revoke-keys" Oct 30 13:08:37.795750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.794908 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5216becf-215f-4045-a425-3bca932690f1" containerName="revoke-keys" Oct 30 13:08:37.795750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.795072 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5216becf-215f-4045-a425-3bca932690f1" containerName="revoke-keys" Oct 30 13:08:37.798914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.798894 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:37.828867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.828838 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s"] Oct 30 13:08:37.830924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.830899 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhqxz\" (UniqueName: \"kubernetes.io/projected/5216becf-215f-4045-a425-3bca932690f1-kube-api-access-qhqxz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:37.830924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.830924 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5216becf-215f-4045-a425-3bca932690f1-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:37.931890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.931861 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:37.931996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.931935 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7knr\" (UniqueName: \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-kube-api-access-d7knr\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:37.931996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.931989 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f9a87d-e4cc-40b4-b079-803e61efbaef-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:37.932098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.932030 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:37.932195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.932168 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:37.932234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.932215 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:37.932283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.932269 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:37.932405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.932390 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:37.932439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.932425 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:37.977936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.977908 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl"] Oct 30 13:08:37.982290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:37.982265 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.027046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.027004 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl"] Oct 30 13:08:38.033449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033414 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.033613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033484 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.033613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033597 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.033731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033634 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7knr\" (UniqueName: \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-kube-api-access-d7knr\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.033731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033690 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f9a87d-e4cc-40b4-b079-803e61efbaef-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.033840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033740 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.033840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033766 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.033840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033803 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.033840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033832 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.034039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033873 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.034039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033898 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.034039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033957 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.034039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.033987 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.034242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.034050 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.034242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.034113 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkmt4\" (UniqueName: \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-kube-api-access-wkmt4\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.034242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.034161 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.034242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.034189 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.034242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.034236 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.034864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.034840 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f9a87d-e4cc-40b4-b079-803e61efbaef-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.035559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.035526 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.035805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.035774 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.035891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.035778 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.036245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.036221 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.037453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.037426 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.038043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.038019 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.060296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.060222 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7knr\" (UniqueName: \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-kube-api-access-d7knr\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.075050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.075025 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.126163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.126135 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:38.135069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135041 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.135182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135137 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.135243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135181 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.135360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135341 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.135462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135374 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.135462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135414 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.135462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135430 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.135638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135560 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.135732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135714 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.135791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135776 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.135859 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.135835 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkmt4\" (UniqueName: \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-kube-api-access-wkmt4\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.136181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.136155 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.136495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.136471 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.136601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.136492 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.138311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.138290 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.138311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.138305 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.170251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.170184 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkmt4\" (UniqueName: \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-kube-api-access-wkmt4\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.171723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.171694 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.297159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.297123 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:38.465412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.465315 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s"] Oct 30 13:08:38.480209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.474984 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.480209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.475117 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.480209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.475160 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.498004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.497970 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" event={"ID":"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef","Type":"ContainerStarted","Data":"2c2291114b0302a794b1fd1b7875d7ba04b270aed78a71966b54d6de73f5eb12"} Oct 30 13:08:38.498136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.498033 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" event={"ID":"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef","Type":"ContainerStarted","Data":"acfffc9b20b91eba6445e857f65c01edda490d13dac32a7831d89f890af24f34"} Oct 30 13:08:38.500007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.499982 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" event={"ID":"5216becf-215f-4045-a425-3bca932690f1","Type":"ContainerDied","Data":"51ca860bda7dd3b88d6ac0fd173ba7d091c5b38f455f438f81bdf6d3d39e7e6c"} Oct 30 13:08:38.504336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.500171 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51ca860bda7dd3b88d6ac0fd173ba7d091c5b38f455f438f81bdf6d3d39e7e6c" Oct 30 13:08:38.504336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.499997 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm" Oct 30 13:08:38.504336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.501669 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" event={"ID":"37f9a87d-e4cc-40b4-b079-803e61efbaef","Type":"ContainerStarted","Data":"ce747ea59a130470890df891c973632edf7e29d19b8758e8df004dfc0b7c6645"} Oct 30 13:08:38.548814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.548754 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" podStartSLOduration=2.548734596 podStartE2EDuration="2.548734596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:38.543530424 +0000 UTC m=+1025.358724846" watchObservedRunningTime="2026-10-30 13:08:38.548734596 +0000 UTC m=+1025.363929019" Oct 30 13:08:38.625657 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:38.625616 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7d2b91_83e5_48e0_9c32_7f2228d2c532.slice/crio-c79e15c20cb5581fa1ef7f234d8e408aa62bca23033cee72519798d949841585 WatchSource:0}: Error finding container c79e15c20cb5581fa1ef7f234d8e408aa62bca23033cee72519798d949841585: Status 404 returned error can't find the container with id c79e15c20cb5581fa1ef7f234d8e408aa62bca23033cee72519798d949841585 Oct 30 13:08:38.626086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.625691 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl"] Oct 30 13:08:38.628642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.628614 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.628738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.628676 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.628738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.628704 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.726192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.726161 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b"] Oct 30 13:08:38.731141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.731106 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.782161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.782127 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b"] Oct 30 13:08:38.847127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.845504 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.847127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.845554 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.847127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.845683 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x45f\" (UniqueName: \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-kube-api-access-5x45f\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.847127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.845726 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.847127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.845858 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.847127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.845963 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.847127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.845991 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.847127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.846036 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.847127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.846151 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.947456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.947418 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.947456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.947466 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.947730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.947515 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.947730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.947630 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.947730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.947717 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.947875 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.947745 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.947875 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.947809 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x45f\" (UniqueName: \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-kube-api-access-5x45f\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.947875 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.947855 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.948022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.947898 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.948022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.947909 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.948247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.948223 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.948828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.948802 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.948986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.948857 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.949146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.949122 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.950708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.950663 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.951616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.951597 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.977633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.977599 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:38.998137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:38.998103 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x45f\" (UniqueName: \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-kube-api-access-5x45f\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:39.075347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.072075 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:39.353555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.345430 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:39.490373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.490342 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b"] Oct 30 13:08:39.490957 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:39.490936 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5efb124_0adc_4302_a97d_2c8f57e42a4a.slice/crio-35f20af8864ca6337760a60a34c57fdd1478e9a1793aef4ce573f62c259c6181 WatchSource:0}: Error finding container 35f20af8864ca6337760a60a34c57fdd1478e9a1793aef4ce573f62c259c6181: Status 404 returned error can't find the container with id 35f20af8864ca6337760a60a34c57fdd1478e9a1793aef4ce573f62c259c6181 Oct 30 13:08:39.493517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.493485 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:39.493603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.493547 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:39.493654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.493592 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:39.581951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.581799 5049 generic.go:356] "Generic (PLEG): container finished" podID="6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef" containerID="2c2291114b0302a794b1fd1b7875d7ba04b270aed78a71966b54d6de73f5eb12" exitCode=0 Oct 30 13:08:39.581951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.581879 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" event={"ID":"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef","Type":"ContainerDied","Data":"2c2291114b0302a794b1fd1b7875d7ba04b270aed78a71966b54d6de73f5eb12"} Oct 30 13:08:39.583649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.583621 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" event={"ID":"e5efb124-0adc-4302-a97d-2c8f57e42a4a","Type":"ContainerStarted","Data":"35f20af8864ca6337760a60a34c57fdd1478e9a1793aef4ce573f62c259c6181"} Oct 30 13:08:39.585239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.585215 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" event={"ID":"37f9a87d-e4cc-40b4-b079-803e61efbaef","Type":"ContainerStarted","Data":"c3f39e5c331dcb049305831e8b8fee1e517f88fec7e14cfdf8c20d1f589c7d69"} Oct 30 13:08:39.588316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.587294 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" event={"ID":"ae7d2b91-83e5-48e0-9c32-7f2228d2c532","Type":"ContainerStarted","Data":"c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f"} Oct 30 13:08:39.588316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.587343 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" event={"ID":"ae7d2b91-83e5-48e0-9c32-7f2228d2c532","Type":"ContainerStarted","Data":"c79e15c20cb5581fa1ef7f234d8e408aa62bca23033cee72519798d949841585"} Oct 30 13:08:39.701618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.701556 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" podStartSLOduration=2.7015401409999997 podStartE2EDuration="2.701540141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:39.683850886 +0000 UTC m=+1026.499045311" watchObservedRunningTime="2026-10-30 13:08:39.701540141 +0000 UTC m=+1026.516734565" Oct 30 13:08:39.767412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:39.767362 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" podStartSLOduration=2.767342668 podStartE2EDuration="2.767342668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:39.764713931 +0000 UTC m=+1026.579908355" watchObservedRunningTime="2026-10-30 13:08:39.767342668 +0000 UTC m=+1026.582537091" Oct 30 13:08:40.135358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:40.131498 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:40.135358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:40.131707 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:08:40.135358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:40.131761 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:08:40.297669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:40.297635 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:40.299228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:40.299201 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:08:40.299446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:40.299424 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:08:40.631603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:40.631567 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" event={"ID":"e5efb124-0adc-4302-a97d-2c8f57e42a4a","Type":"ContainerStarted","Data":"3e5cfbb7c8ba1d78767353ead0240526f7d392de149f3141aa4af23548f980f4"} Oct 30 13:08:40.697792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:40.697731 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" podStartSLOduration=2.697707117 podStartE2EDuration="2.697707117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:40.688796823 +0000 UTC m=+1027.503991287" watchObservedRunningTime="2026-10-30 13:08:40.697707117 +0000 UTC m=+1027.512901543" Oct 30 13:08:41.088238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.088205 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:41.088931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.088848 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:08:41.090448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.090416 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 13:08:41.377367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.373966 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:08:41.377367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.374038 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:08:41.392355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.379401 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:08:41.392355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.379467 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:08:41.472205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.471825 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:41.536346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.529912 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-openshift-ca\") pod \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " Oct 30 13:08:41.536346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.530010 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-xks-ca\") pod \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " Oct 30 13:08:41.536346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.530082 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-kube-api-access-8q8s6\" (UniqueName: \"kubernetes.io/projected/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-kube-api-access-8q8s6\") pod \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\" (UID: \"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef\") " Oct 30 13:08:41.600348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.566584 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-openshift-ca" pod "6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef" (UID: "6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:41.691651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.691566 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" Oct 30 13:08:41.691959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.691933 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj" event={"ID":"6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef","Type":"ContainerDied","Data":"acfffc9b20b91eba6445e857f65c01edda490d13dac32a7831d89f890af24f34"} Oct 30 13:08:41.692097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.692081 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acfffc9b20b91eba6445e857f65c01edda490d13dac32a7831d89f890af24f34" Oct 30 13:08:41.703337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.703293 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.774003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.773963 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-kube-api-access-8q8s6" pod "6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef" (UID: "6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef"). InnerVolumeSpecName "kube-api-access-8q8s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:41.825352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.820876 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q8s6\" (UniqueName: \"kubernetes.io/projected/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-kube-api-access-8q8s6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.905361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.904808 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:41.964407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:41.962263 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-xks-ca" pod "6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef" (UID: "6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:42.047345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:42.038552 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:42.084530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:42.074913 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:08:42.084530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:42.074969 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 13:08:42.084530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:42.076121 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:42.151368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:42.150945 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:08:42.151368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:42.151005 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:08:42.316098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:42.305460 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:08:42.323456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:42.323404 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:08:43.081427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:43.081375 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:08:43.081731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:43.081685 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 13:08:43.146346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:43.135076 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:08:43.146346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:43.135135 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:08:43.309479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:43.309428 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:08:43.315433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:43.315389 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:08:44.083370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:44.083313 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:08:44.084045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:44.084005 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 13:08:44.153392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:44.148747 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:08:44.153392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:44.148824 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:08:44.299578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:44.299534 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:08:44.303343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:44.299945 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:08:45.077581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:45.077111 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:08:45.077581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:45.077175 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 13:08:45.154956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:45.154916 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 13:08:45.155220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:45.155179 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 13:08:45.297469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:45.297435 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:08:45.297723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:45.297697 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 13:08:45.489636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:45.489544 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:46.088345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:46.084065 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:08:46.088345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:46.084128 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 13:08:46.303516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:46.303476 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:46.304355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:46.304317 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:46.305124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:46.305100 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:08:46.441844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:46.441747 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:46.442463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:46.442440 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:46.444031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:46.443983 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:08:47.072855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:47.072820 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:08:47.073032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:47.072888 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 13:08:48.078311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.078272 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:48.078719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.078683 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:48.079279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.079258 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:08:48.346364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.346232 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:48.693364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.693264 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p"] Oct 30 13:08:48.694027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.694004 5049 cpu_manager.go:414] "containerMap: removing container" podUID="6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef" containerName="revoke-keys" Oct 30 13:08:48.694146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.694029 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef" containerName="revoke-keys" Oct 30 13:08:48.694245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.694231 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef" containerName="revoke-keys" Oct 30 13:08:48.698479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.698459 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:48.713921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.713895 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:48.720267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.720245 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:08:48.735035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.735003 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p"] Oct 30 13:08:48.853136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.853092 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b77dbc3d-jj26p\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:48.853307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.853227 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b77dbc3d-jj26p\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:48.853307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.853297 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8z6\" (UniqueName: \"kubernetes.io/projected/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-kube-api-access-ck8z6\") pod \"maas-api-revoke-sub-b77dbc3d-jj26p\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:48.954960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.954862 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b77dbc3d-jj26p\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:48.955135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.954955 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8z6\" (UniqueName: \"kubernetes.io/projected/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-kube-api-access-ck8z6\") pod \"maas-api-revoke-sub-b77dbc3d-jj26p\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:48.955196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.955158 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b77dbc3d-jj26p\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:48.955575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.955547 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b77dbc3d-jj26p\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:48.957800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.957762 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b77dbc3d-jj26p\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:48.980711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:48.980682 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8z6\" (UniqueName: \"kubernetes.io/projected/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-kube-api-access-ck8z6\") pod \"maas-api-revoke-sub-b77dbc3d-jj26p\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:49.014099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:49.014068 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:49.358471 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:49.354891 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040c860d_6c32_4fbc_ba13_56cf5a84ab5c.slice/crio-0fd5de0ca9ce9f343a336ff6777bc866528b4b4d3f4f74e7fe5684114fa7eb85 WatchSource:0}: Error finding container 0fd5de0ca9ce9f343a336ff6777bc866528b4b4d3f4f74e7fe5684114fa7eb85: Status 404 returned error can't find the container with id 0fd5de0ca9ce9f343a336ff6777bc866528b4b4d3f4f74e7fe5684114fa7eb85 Oct 30 13:08:49.358471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:49.357801 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p"] Oct 30 13:08:49.942275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:49.942236 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" event={"ID":"040c860d-6c32-4fbc-ba13-56cf5a84ab5c","Type":"ContainerStarted","Data":"ca7d80395598890947df0ee1f11e2b256a875e33a294efceeefef1925264cb0e"} Oct 30 13:08:49.942485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:49.942294 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" event={"ID":"040c860d-6c32-4fbc-ba13-56cf5a84ab5c","Type":"ContainerStarted","Data":"0fd5de0ca9ce9f343a336ff6777bc866528b4b4d3f4f74e7fe5684114fa7eb85"} Oct 30 13:08:49.989400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:49.989256 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" podStartSLOduration=1.989239962 podStartE2EDuration="1.989239962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:49.986865293 +0000 UTC m=+1036.802059716" watchObservedRunningTime="2026-10-30 13:08:49.989239962 +0000 UTC m=+1036.804434385" Oct 30 13:08:50.172593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:50.172548 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:50.324708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:50.324652 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:50.501829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:50.501797 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:50.687505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:50.687422 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:50.955017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:50.954919 5049 generic.go:356] "Generic (PLEG): container finished" podID="040c860d-6c32-4fbc-ba13-56cf5a84ab5c" containerID="ca7d80395598890947df0ee1f11e2b256a875e33a294efceeefef1925264cb0e" exitCode=0 Oct 30 13:08:50.955017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:50.954997 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" event={"ID":"040c860d-6c32-4fbc-ba13-56cf5a84ab5c","Type":"ContainerDied","Data":"ca7d80395598890947df0ee1f11e2b256a875e33a294efceeefef1925264cb0e"} Oct 30 13:08:52.138437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.138411 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:52.177175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.177143 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72"] Oct 30 13:08:52.177802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.177789 5049 cpu_manager.go:414] "containerMap: removing container" podUID="040c860d-6c32-4fbc-ba13-56cf5a84ab5c" containerName="revoke-keys" Oct 30 13:08:52.177860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.177804 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040c860d-6c32-4fbc-ba13-56cf5a84ab5c" containerName="revoke-keys" Oct 30 13:08:52.177928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.177918 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="040c860d-6c32-4fbc-ba13-56cf5a84ab5c" containerName="revoke-keys" Oct 30 13:08:52.181640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.181620 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.210436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.210403 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72"] Oct 30 13:08:52.306782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.306740 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-openshift-ca\") pod \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " Oct 30 13:08:52.306959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.306807 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-kube-api-access-ck8z6\" (UniqueName: \"kubernetes.io/projected/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-kube-api-access-ck8z6\") pod \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " Oct 30 13:08:52.307056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.306969 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-xks-ca\") pod \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\" (UID: \"040c860d-6c32-4fbc-ba13-56cf5a84ab5c\") " Oct 30 13:08:52.307197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.307160 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-openshift-ca" pod "040c860d-6c32-4fbc-ba13-56cf5a84ab5c" (UID: "040c860d-6c32-4fbc-ba13-56cf5a84ab5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:52.307760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.307729 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff7dcd08-llg72\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.307903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.307878 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nr6j\" (UniqueName: \"kubernetes.io/projected/a60a574d-b990-4931-9b15-f0fc5741aa3e-kube-api-access-9nr6j\") pod \"maas-api-revoke-sub-ff7dcd08-llg72\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.307995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.307981 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff7dcd08-llg72\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.308145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.308130 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:52.309447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.309423 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-kube-api-access-ck8z6" pod "040c860d-6c32-4fbc-ba13-56cf5a84ab5c" (UID: "040c860d-6c32-4fbc-ba13-56cf5a84ab5c"). InnerVolumeSpecName "kube-api-access-ck8z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:52.309673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.309656 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-xks-ca" pod "040c860d-6c32-4fbc-ba13-56cf5a84ab5c" (UID: "040c860d-6c32-4fbc-ba13-56cf5a84ab5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:52.409406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.409365 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nr6j\" (UniqueName: \"kubernetes.io/projected/a60a574d-b990-4931-9b15-f0fc5741aa3e-kube-api-access-9nr6j\") pod \"maas-api-revoke-sub-ff7dcd08-llg72\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.409599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.409494 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff7dcd08-llg72\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.409772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.409747 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff7dcd08-llg72\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.409852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.409832 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck8z6\" (UniqueName: \"kubernetes.io/projected/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-kube-api-access-ck8z6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:52.409901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.409852 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040c860d-6c32-4fbc-ba13-56cf5a84ab5c-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:52.411102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.411051 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff7dcd08-llg72\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.416377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.413852 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff7dcd08-llg72\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.439130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.439086 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nr6j\" (UniqueName: \"kubernetes.io/projected/a60a574d-b990-4931-9b15-f0fc5741aa3e-kube-api-access-9nr6j\") pod \"maas-api-revoke-sub-ff7dcd08-llg72\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.513238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.513193 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:52.525423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.525387 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:52.839349 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:08:52.839289 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60a574d_b990_4931_9b15_f0fc5741aa3e.slice/crio-6fb8cb24db870e22cea72b101998a5779543f6760383de501d068399d9d093f9 WatchSource:0}: Error finding container 6fb8cb24db870e22cea72b101998a5779543f6760383de501d068399d9d093f9: Status 404 returned error can't find the container with id 6fb8cb24db870e22cea72b101998a5779543f6760383de501d068399d9d093f9 Oct 30 13:08:52.841120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.841087 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72"] Oct 30 13:08:52.973357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.973292 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" Oct 30 13:08:52.973357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.973306 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p" event={"ID":"040c860d-6c32-4fbc-ba13-56cf5a84ab5c","Type":"ContainerDied","Data":"0fd5de0ca9ce9f343a336ff6777bc866528b4b4d3f4f74e7fe5684114fa7eb85"} Oct 30 13:08:52.973605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.973399 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd5de0ca9ce9f343a336ff6777bc866528b4b4d3f4f74e7fe5684114fa7eb85" Oct 30 13:08:52.975291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:52.975262 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" event={"ID":"a60a574d-b990-4931-9b15-f0fc5741aa3e","Type":"ContainerStarted","Data":"6fb8cb24db870e22cea72b101998a5779543f6760383de501d068399d9d093f9"} Oct 30 13:08:54.081436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:54.070707 5049 generic.go:356] "Generic (PLEG): container finished" podID="a60a574d-b990-4931-9b15-f0fc5741aa3e" containerID="41645b56790f6f831211acf0e24a78b1e66d59dc7603101ebfc699072a5ba0c8" exitCode=0 Oct 30 13:08:54.081436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:54.070988 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" event={"ID":"a60a574d-b990-4931-9b15-f0fc5741aa3e","Type":"ContainerDied","Data":"41645b56790f6f831211acf0e24a78b1e66d59dc7603101ebfc699072a5ba0c8"} Oct 30 13:08:54.451394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:54.450815 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:55.584345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:55.583436 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:55.584905 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:55.584630 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:55.608346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:55.607369 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:55.785685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:55.785642 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:08:55.786006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:55.785963 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:55.786431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:55.785917 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:55.812722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:55.812363 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:08:56.244752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.183859 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.244752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.203618 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.296346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.288988 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": context deadline exceeded" Oct 30 13:08:56.584964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.584901 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:08:56.608383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.593978 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:08:56.617370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.608677 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:08:56.629454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.617689 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:08:56.791415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.738899 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.791415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.747459 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:08:56.791415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.748400 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.829347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.805353 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.893927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.812414 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.893927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.856100 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:08:56.893927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.856168 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:08:56.893927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.826390 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:08:56.893927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.832809 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.893927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.858502 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:08:56.893927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:56.858556 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:08:57.090940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.090907 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:57.235671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.184415 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.235671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.222643 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.235671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.223165 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:08:57.286374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.260044 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:08:57.286374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.260119 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:08:57.333074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.289287 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60a574d-b990-4931-9b15-f0fc5741aa3e-kube-api-access-9nr6j\" (UniqueName: \"kubernetes.io/projected/a60a574d-b990-4931-9b15-f0fc5741aa3e-kube-api-access-9nr6j\") pod \"a60a574d-b990-4931-9b15-f0fc5741aa3e\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " Oct 30 13:08:57.333074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.289426 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-xks-ca\") pod \"a60a574d-b990-4931-9b15-f0fc5741aa3e\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " Oct 30 13:08:57.333074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.289507 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-openshift-ca\") pod \"a60a574d-b990-4931-9b15-f0fc5741aa3e\" (UID: \"a60a574d-b990-4931-9b15-f0fc5741aa3e\") " Oct 30 13:08:57.333074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.323544 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-openshift-ca" pod "a60a574d-b990-4931-9b15-f0fc5741aa3e" (UID: "a60a574d-b990-4931-9b15-f0fc5741aa3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:57.445899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.415795 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:57.516398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.479751 5049 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:57.516398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.479812 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.517238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.517205 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60a574d-b990-4931-9b15-f0fc5741aa3e-kube-api-access-9nr6j" pod "a60a574d-b990-4931-9b15-f0fc5741aa3e" (UID: "a60a574d-b990-4931-9b15-f0fc5741aa3e"). InnerVolumeSpecName "kube-api-access-9nr6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:57.577608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.577562 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nr6j\" (UniqueName: \"kubernetes.io/projected/a60a574d-b990-4931-9b15-f0fc5741aa3e-kube-api-access-9nr6j\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:57.580582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.580549 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-xks-ca" pod "a60a574d-b990-4931-9b15-f0fc5741aa3e" (UID: "a60a574d-b990-4931-9b15-f0fc5741aa3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:57.635346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.615122 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:08:57.835390 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:08:57.824849 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:08:57.910585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.875052 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a574d-b990-4931-9b15-f0fc5741aa3e-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:08:57.988366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.896744 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:08:58.033561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:57.896793 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 13:08:58.095408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:58.077532 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:08:58.095712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:58.095672 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:08:58.212867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:58.202473 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:08:58.257240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:58.230374 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:08:58.257240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:58.230516 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:58.373178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:58.345374 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" Oct 30 13:08:58.373178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:58.357678 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:58.373178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:58.358420 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72" event={"ID":"a60a574d-b990-4931-9b15-f0fc5741aa3e","Type":"ContainerDied","Data":"6fb8cb24db870e22cea72b101998a5779543f6760383de501d068399d9d093f9"} Oct 30 13:08:58.373178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:58.358495 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fb8cb24db870e22cea72b101998a5779543f6760383de501d068399d9d093f9" Oct 30 13:08:58.615543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:58.615478 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.074367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.058403 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:08:59.074367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.058800 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:08:59.074367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.059067 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 13:08:59.074367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.073410 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.107391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.106614 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.213347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.210443 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:08:59.246156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.244196 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.246156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.244570 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:08:59.359510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.359402 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:08:59.640673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.640563 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.797356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.794902 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:08:59.797356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.795264 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:08:59.912455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.910252 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:08:59.947258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.945222 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:08:59.994614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:08:59.994586 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:09:00.040677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.040631 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:09:00.059058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.059007 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:00.081352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.080149 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:09:00.132637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.132609 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:09:00.185463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.185382 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g"] Oct 30 13:09:00.211177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.211134 5049 cpu_manager.go:414] "containerMap: removing container" podUID="a60a574d-b990-4931-9b15-f0fc5741aa3e" containerName="revoke-keys" Oct 30 13:09:00.229388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.228296 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60a574d-b990-4931-9b15-f0fc5741aa3e" containerName="revoke-keys" Oct 30 13:09:00.229388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.228665 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60a574d-b990-4931-9b15-f0fc5741aa3e" containerName="revoke-keys" Oct 30 13:09:00.236148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.234970 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g"] Oct 30 13:09:00.236148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.235081 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.258761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.253231 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:09:00.258761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.253305 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:09:00.284134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.284101 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:00.378990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.378954 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d89fa0d-tpp7g\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.379189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.379099 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx9nf\" (UniqueName: \"kubernetes.io/projected/e86ea149-96af-40ef-aef9-b54ec08ef356-kube-api-access-gx9nf\") pod \"maas-api-revoke-sub-3d89fa0d-tpp7g\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.379295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.379271 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d89fa0d-tpp7g\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.480843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.480270 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx9nf\" (UniqueName: \"kubernetes.io/projected/e86ea149-96af-40ef-aef9-b54ec08ef356-kube-api-access-gx9nf\") pod \"maas-api-revoke-sub-3d89fa0d-tpp7g\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.480843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.480475 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d89fa0d-tpp7g\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.480843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.480651 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d89fa0d-tpp7g\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.481616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.481590 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d89fa0d-tpp7g\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.483834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.483806 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d89fa0d-tpp7g\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.506741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.506710 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx9nf\" (UniqueName: \"kubernetes.io/projected/e86ea149-96af-40ef-aef9-b54ec08ef356-kube-api-access-gx9nf\") pod \"maas-api-revoke-sub-3d89fa0d-tpp7g\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.574722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.574690 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:00.904930 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:09:00.904890 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86ea149_96af_40ef_aef9_b54ec08ef356.slice/crio-a9eafbd59cb1a88f6f7a614dfc9644c2ae7d0a99bb66ba5fce0b03956cc1849f WatchSource:0}: Error finding container a9eafbd59cb1a88f6f7a614dfc9644c2ae7d0a99bb66ba5fce0b03956cc1849f: Status 404 returned error can't find the container with id a9eafbd59cb1a88f6f7a614dfc9644c2ae7d0a99bb66ba5fce0b03956cc1849f Oct 30 13:09:00.905692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:00.905666 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g"] Oct 30 13:09:01.501067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:01.500983 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" event={"ID":"e86ea149-96af-40ef-aef9-b54ec08ef356","Type":"ContainerStarted","Data":"d55de42f80899cd9caf62da018acf0e5643cf737bf5dc8a11ad6520e9c4bcf76"} Oct 30 13:09:01.501067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:01.501036 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" event={"ID":"e86ea149-96af-40ef-aef9-b54ec08ef356","Type":"ContainerStarted","Data":"a9eafbd59cb1a88f6f7a614dfc9644c2ae7d0a99bb66ba5fce0b03956cc1849f"} Oct 30 13:09:01.542248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:01.542180 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" podStartSLOduration=1.542162669 podStartE2EDuration="1.542162669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:01.541699386 +0000 UTC m=+1048.356893810" watchObservedRunningTime="2026-10-30 13:09:01.542162669 +0000 UTC m=+1048.357357093" Oct 30 13:09:02.398146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:02.398113 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:02.519631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:02.519597 5049 generic.go:356] "Generic (PLEG): container finished" podID="e86ea149-96af-40ef-aef9-b54ec08ef356" containerID="d55de42f80899cd9caf62da018acf0e5643cf737bf5dc8a11ad6520e9c4bcf76" exitCode=0 Oct 30 13:09:02.520124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:02.519740 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" event={"ID":"e86ea149-96af-40ef-aef9-b54ec08ef356","Type":"ContainerDied","Data":"d55de42f80899cd9caf62da018acf0e5643cf737bf5dc8a11ad6520e9c4bcf76"} Oct 30 13:09:03.707425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.707075 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:03.724380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.724350 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-xks-ca\") pod \"e86ea149-96af-40ef-aef9-b54ec08ef356\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " Oct 30 13:09:03.724537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.724510 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86ea149-96af-40ef-aef9-b54ec08ef356-kube-api-access-gx9nf\" (UniqueName: \"kubernetes.io/projected/e86ea149-96af-40ef-aef9-b54ec08ef356-kube-api-access-gx9nf\") pod \"e86ea149-96af-40ef-aef9-b54ec08ef356\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " Oct 30 13:09:03.724603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.724590 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-openshift-ca\") pod \"e86ea149-96af-40ef-aef9-b54ec08ef356\" (UID: \"e86ea149-96af-40ef-aef9-b54ec08ef356\") " Oct 30 13:09:03.725007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.724975 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-openshift-ca" pod "e86ea149-96af-40ef-aef9-b54ec08ef356" (UID: "e86ea149-96af-40ef-aef9-b54ec08ef356"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:03.725279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.725260 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:03.726415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.726391 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-xks-ca" pod "e86ea149-96af-40ef-aef9-b54ec08ef356" (UID: "e86ea149-96af-40ef-aef9-b54ec08ef356"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:03.726947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.726913 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86ea149-96af-40ef-aef9-b54ec08ef356-kube-api-access-gx9nf" pod "e86ea149-96af-40ef-aef9-b54ec08ef356" (UID: "e86ea149-96af-40ef-aef9-b54ec08ef356"). InnerVolumeSpecName "kube-api-access-gx9nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:03.822802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.822765 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk"] Oct 30 13:09:03.823484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.823464 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e86ea149-96af-40ef-aef9-b54ec08ef356" containerName="revoke-keys" Oct 30 13:09:03.823484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.823483 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86ea149-96af-40ef-aef9-b54ec08ef356" containerName="revoke-keys" Oct 30 13:09:03.823645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.823614 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86ea149-96af-40ef-aef9-b54ec08ef356" containerName="revoke-keys" Oct 30 13:09:03.826340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.826288 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx9nf\" (UniqueName: \"kubernetes.io/projected/e86ea149-96af-40ef-aef9-b54ec08ef356-kube-api-access-gx9nf\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:03.826447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.826360 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86ea149-96af-40ef-aef9-b54ec08ef356-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:03.827314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.827295 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:03.867687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.867650 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk"] Oct 30 13:09:03.927879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.927839 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a0264e-kjmvk\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:03.928048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.927983 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a0264e-kjmvk\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:03.928179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:03.928146 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r6hx\" (UniqueName: \"kubernetes.io/projected/7087d8e3-8fc2-4985-b20b-15c650d53368-kube-api-access-9r6hx\") pod \"maas-api-revoke-sub-e0a0264e-kjmvk\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:04.029425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.029265 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6hx\" (UniqueName: \"kubernetes.io/projected/7087d8e3-8fc2-4985-b20b-15c650d53368-kube-api-access-9r6hx\") pod \"maas-api-revoke-sub-e0a0264e-kjmvk\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:04.029425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.029411 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a0264e-kjmvk\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:04.044305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.029505 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a0264e-kjmvk\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:04.044305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.030069 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a0264e-kjmvk\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:04.044305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.032049 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a0264e-kjmvk\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:04.053110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.053086 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6hx\" (UniqueName: \"kubernetes.io/projected/7087d8e3-8fc2-4985-b20b-15c650d53368-kube-api-access-9r6hx\") pod \"maas-api-revoke-sub-e0a0264e-kjmvk\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:04.154368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.154280 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:04.480702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.480674 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk"] Oct 30 13:09:04.481053 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:09:04.481027 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7087d8e3_8fc2_4985_b20b_15c650d53368.slice/crio-225984d736fa4bcff7ec30ed03ffa9662d53408c2b266b100d5799826da27128 WatchSource:0}: Error finding container 225984d736fa4bcff7ec30ed03ffa9662d53408c2b266b100d5799826da27128: Status 404 returned error can't find the container with id 225984d736fa4bcff7ec30ed03ffa9662d53408c2b266b100d5799826da27128 Oct 30 13:09:04.512114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.512080 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:04.545453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.545397 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" event={"ID":"7087d8e3-8fc2-4985-b20b-15c650d53368","Type":"ContainerStarted","Data":"225984d736fa4bcff7ec30ed03ffa9662d53408c2b266b100d5799826da27128"} Oct 30 13:09:04.547506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.547455 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" event={"ID":"e86ea149-96af-40ef-aef9-b54ec08ef356","Type":"ContainerDied","Data":"a9eafbd59cb1a88f6f7a614dfc9644c2ae7d0a99bb66ba5fce0b03956cc1849f"} Oct 30 13:09:04.547694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.547514 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g" Oct 30 13:09:04.547694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:04.547530 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9eafbd59cb1a88f6f7a614dfc9644c2ae7d0a99bb66ba5fce0b03956cc1849f" Oct 30 13:09:05.558717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:05.558686 5049 generic.go:356] "Generic (PLEG): container finished" podID="7087d8e3-8fc2-4985-b20b-15c650d53368" containerID="10cdd6da4083b29033353c4e0e9acd5c0cfe50f11ac96e2625bf607fc53e639e" exitCode=0 Oct 30 13:09:05.559080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:05.558752 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" event={"ID":"7087d8e3-8fc2-4985-b20b-15c650d53368","Type":"ContainerDied","Data":"10cdd6da4083b29033353c4e0e9acd5c0cfe50f11ac96e2625bf607fc53e639e"} Oct 30 13:09:06.334940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.331711 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:06.724475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.724453 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:06.766937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.766903 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-xks-ca\") pod \"7087d8e3-8fc2-4985-b20b-15c650d53368\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " Oct 30 13:09:06.767218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.767196 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7087d8e3-8fc2-4985-b20b-15c650d53368-kube-api-access-9r6hx\" (UniqueName: \"kubernetes.io/projected/7087d8e3-8fc2-4985-b20b-15c650d53368-kube-api-access-9r6hx\") pod \"7087d8e3-8fc2-4985-b20b-15c650d53368\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " Oct 30 13:09:06.767310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.767251 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-openshift-ca\") pod \"7087d8e3-8fc2-4985-b20b-15c650d53368\" (UID: \"7087d8e3-8fc2-4985-b20b-15c650d53368\") " Oct 30 13:09:06.767974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.767946 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-openshift-ca" pod "7087d8e3-8fc2-4985-b20b-15c650d53368" (UID: "7087d8e3-8fc2-4985-b20b-15c650d53368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:06.768271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.768247 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:06.769580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.769556 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-xks-ca" pod "7087d8e3-8fc2-4985-b20b-15c650d53368" (UID: "7087d8e3-8fc2-4985-b20b-15c650d53368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:06.769870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.769849 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7087d8e3-8fc2-4985-b20b-15c650d53368-kube-api-access-9r6hx" pod "7087d8e3-8fc2-4985-b20b-15c650d53368" (UID: "7087d8e3-8fc2-4985-b20b-15c650d53368"). InnerVolumeSpecName "kube-api-access-9r6hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:06.869189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.869155 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7087d8e3-8fc2-4985-b20b-15c650d53368-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:06.869189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:06.869186 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r6hx\" (UniqueName: \"kubernetes.io/projected/7087d8e3-8fc2-4985-b20b-15c650d53368-kube-api-access-9r6hx\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:07.576351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:07.576285 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" Oct 30 13:09:07.576351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:07.576297 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk" event={"ID":"7087d8e3-8fc2-4985-b20b-15c650d53368","Type":"ContainerDied","Data":"225984d736fa4bcff7ec30ed03ffa9662d53408c2b266b100d5799826da27128"} Oct 30 13:09:07.576571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:07.576369 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="225984d736fa4bcff7ec30ed03ffa9662d53408c2b266b100d5799826da27128" Oct 30 13:09:08.194685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:08.194651 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:10.009302 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:10.009196 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:11.940607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:11.940566 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:13.467282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.467244 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn"] Oct 30 13:09:13.467952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.467936 5049 cpu_manager.go:414] "containerMap: removing container" podUID="7087d8e3-8fc2-4985-b20b-15c650d53368" containerName="revoke-keys" Oct 30 13:09:13.467994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.467953 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7087d8e3-8fc2-4985-b20b-15c650d53368" containerName="revoke-keys" Oct 30 13:09:13.468069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.468060 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="7087d8e3-8fc2-4985-b20b-15c650d53368" containerName="revoke-keys" Oct 30 13:09:13.473025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.473007 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:13.496154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.496120 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn"] Oct 30 13:09:13.501601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.501572 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:09:13.501772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.501603 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:09:13.568935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.568905 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50d7e814-wcppn\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:13.569089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.569033 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjzf6\" (UniqueName: \"kubernetes.io/projected/b2c1c19d-c997-4399-9f2c-75092707fc86-kube-api-access-jjzf6\") pod \"maas-api-revoke-sub-50d7e814-wcppn\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:13.569128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.569086 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50d7e814-wcppn\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:13.670220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.670185 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50d7e814-wcppn\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:13.670458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.670299 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjzf6\" (UniqueName: \"kubernetes.io/projected/b2c1c19d-c997-4399-9f2c-75092707fc86-kube-api-access-jjzf6\") pod \"maas-api-revoke-sub-50d7e814-wcppn\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:13.670458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.670407 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50d7e814-wcppn\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:13.670995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.670967 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50d7e814-wcppn\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:13.672788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.672766 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50d7e814-wcppn\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:13.694219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.694193 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjzf6\" (UniqueName: \"kubernetes.io/projected/b2c1c19d-c997-4399-9f2c-75092707fc86-kube-api-access-jjzf6\") pod \"maas-api-revoke-sub-50d7e814-wcppn\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:13.787166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:13.787087 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:14.023188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:14.023142 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:14.116433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:14.116395 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn"] Oct 30 13:09:14.636919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:14.636881 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" event={"ID":"b2c1c19d-c997-4399-9f2c-75092707fc86","Type":"ContainerStarted","Data":"60f7d6cdf28a30375beaeb5181cbea8b79c81c93d570a0d9ef0e5025b79fb2ef"} Oct 30 13:09:14.637314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:14.636954 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" event={"ID":"b2c1c19d-c997-4399-9f2c-75092707fc86","Type":"ContainerStarted","Data":"013dd6b8fb9b0e2d023c80693af1d49e843abb58e88408038aea76f150ed0111"} Oct 30 13:09:14.685299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:14.685209 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" podStartSLOduration=1.685196047 podStartE2EDuration="1.685196047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:14.682553595 +0000 UTC m=+1061.497748018" watchObservedRunningTime="2026-10-30 13:09:14.685196047 +0000 UTC m=+1061.500390528" Oct 30 13:09:15.647624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:15.647585 5049 generic.go:356] "Generic (PLEG): container finished" podID="b2c1c19d-c997-4399-9f2c-75092707fc86" containerID="60f7d6cdf28a30375beaeb5181cbea8b79c81c93d570a0d9ef0e5025b79fb2ef" exitCode=0 Oct 30 13:09:15.648026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:15.647666 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" event={"ID":"b2c1c19d-c997-4399-9f2c-75092707fc86","Type":"ContainerDied","Data":"60f7d6cdf28a30375beaeb5181cbea8b79c81c93d570a0d9ef0e5025b79fb2ef"} Oct 30 13:09:15.980425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:15.979291 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:16.830129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:16.830101 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:16.912970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:16.912867 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c1c19d-c997-4399-9f2c-75092707fc86-kube-api-access-jjzf6\" (UniqueName: \"kubernetes.io/projected/b2c1c19d-c997-4399-9f2c-75092707fc86-kube-api-access-jjzf6\") pod \"b2c1c19d-c997-4399-9f2c-75092707fc86\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " Oct 30 13:09:16.913159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:16.913011 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-openshift-ca\") pod \"b2c1c19d-c997-4399-9f2c-75092707fc86\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " Oct 30 13:09:16.913159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:16.913122 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-xks-ca\") pod \"b2c1c19d-c997-4399-9f2c-75092707fc86\" (UID: \"b2c1c19d-c997-4399-9f2c-75092707fc86\") " Oct 30 13:09:16.913675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:16.913643 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-openshift-ca" pod "b2c1c19d-c997-4399-9f2c-75092707fc86" (UID: "b2c1c19d-c997-4399-9f2c-75092707fc86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:16.913860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:16.913834 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:16.917632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:16.917603 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c1c19d-c997-4399-9f2c-75092707fc86-kube-api-access-jjzf6" pod "b2c1c19d-c997-4399-9f2c-75092707fc86" (UID: "b2c1c19d-c997-4399-9f2c-75092707fc86"). InnerVolumeSpecName "kube-api-access-jjzf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:16.917949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:16.917917 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-xks-ca" pod "b2c1c19d-c997-4399-9f2c-75092707fc86" (UID: "b2c1c19d-c997-4399-9f2c-75092707fc86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:17.015856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:17.015816 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjzf6\" (UniqueName: \"kubernetes.io/projected/b2c1c19d-c997-4399-9f2c-75092707fc86-kube-api-access-jjzf6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:17.015856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:17.015854 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c1c19d-c997-4399-9f2c-75092707fc86-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:17.666348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:17.666256 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" Oct 30 13:09:17.666348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:17.666252 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn" event={"ID":"b2c1c19d-c997-4399-9f2c-75092707fc86","Type":"ContainerDied","Data":"013dd6b8fb9b0e2d023c80693af1d49e843abb58e88408038aea76f150ed0111"} Oct 30 13:09:17.666526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:17.666361 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="013dd6b8fb9b0e2d023c80693af1d49e843abb58e88408038aea76f150ed0111" Oct 30 13:09:17.849427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:17.849397 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:19.083028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:19.082977 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:20.088345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:20.088002 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:21.792577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:21.791394 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:22.041391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:22.024239 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:22.072434 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:22.063411 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:22.117463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:22.117417 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:22.189038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:22.188400 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:22.993461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:22.993410 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:23.029350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.028411 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:23.029350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.029197 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:09:23.078383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.063724 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:23.078383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.064499 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:09:23.208342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.191397 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:23.208342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.192815 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:09:23.208342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.205402 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:23.208342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.205875 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:09:23.341009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.340962 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:09:23.371343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.359780 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:09:23.614606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.577870 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:09:23.614606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.607468 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:23.793484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.793441 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:23.810509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.810473 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:09:23.843970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.843936 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:09:23.909835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.909759 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s"] Oct 30 13:09:23.974176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.940751 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b2c1c19d-c997-4399-9f2c-75092707fc86" containerName="revoke-keys" Oct 30 13:09:23.982495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.982467 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c1c19d-c997-4399-9f2c-75092707fc86" containerName="revoke-keys" Oct 30 13:09:23.982956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.982933 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c1c19d-c997-4399-9f2c-75092707fc86" containerName="revoke-keys" Oct 30 13:09:23.998527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.998488 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:09:24.021354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:23.955763 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:24.064448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.064405 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:24.201562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.201459 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:24.266289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.265971 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.267352 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.267386 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s"] Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.267424 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.268839 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.277377 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.277427 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.277458 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdj9g\" (UniqueName: \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-kube-api-access-mdj9g\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.277527 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.277573 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.277598 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.277751 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.277798 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.279361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.277823 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.299829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.299800 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:09:24.309619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.309592 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g"] Oct 30 13:09:24.314538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.314515 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr"] Oct 30 13:09:24.314669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.314640 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.318870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.318850 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.318985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.318888 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g"] Oct 30 13:09:24.324018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.323981 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr"] Oct 30 13:09:24.334720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.334574 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:09:24.334720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.334650 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:09:24.379177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.379140 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.379400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.379208 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.379400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.379232 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.379541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.379517 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.379909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.379885 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.380026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.379957 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.380154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.380131 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.380206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.380189 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.381150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.380631 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.381150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.380697 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.381150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.380914 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdj9g\" (UniqueName: \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-kube-api-access-mdj9g\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.381150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.380970 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.381150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.381080 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.381150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.381090 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.382236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.382216 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.383276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.383255 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.412713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.412678 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.413226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.413198 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdj9g\" (UniqueName: \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-kube-api-access-mdj9g\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.482674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.482557 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.482674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.482641 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.482990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.482738 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.482990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.482799 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5jz\" (UniqueName: \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-kube-api-access-dm5jz\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.482990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.482835 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.482990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.482861 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b70acc-a40d-4065-8c58-53a6fff7d969-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.482990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.482889 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aa9e2f8-rsl6g\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.482990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.482933 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.483295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.482990 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.483295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.483018 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.483295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.483057 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aa9e2f8-rsl6g\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.483295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.483097 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4xxj\" (UniqueName: \"kubernetes.io/projected/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-kube-api-access-r4xxj\") pod \"maas-api-revoke-sub-8aa9e2f8-rsl6g\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.584682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.584647 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.584860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.584730 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5jz\" (UniqueName: \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-kube-api-access-dm5jz\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.584860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.584765 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.584860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.584796 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b70acc-a40d-4065-8c58-53a6fff7d969-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.584860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.584829 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aa9e2f8-rsl6g\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.585296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.584875 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.585296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.584937 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.585296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.584968 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.585296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.585007 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aa9e2f8-rsl6g\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.585296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.585042 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4xxj\" (UniqueName: \"kubernetes.io/projected/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-kube-api-access-r4xxj\") pod \"maas-api-revoke-sub-8aa9e2f8-rsl6g\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.585296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.585065 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.585296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.585154 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.585296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.585169 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.585296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.585247 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.585296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.585257 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.585829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.585309 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.585829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.585618 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b70acc-a40d-4065-8c58-53a6fff7d969-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.585829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.585716 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aa9e2f8-rsl6g\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.587449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.587425 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.587664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.587647 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aa9e2f8-rsl6g\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.587741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.587663 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.605292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.605272 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:24.610076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.609601 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5jz\" (UniqueName: \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-kube-api-access-dm5jz\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.613989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.613964 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.614145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.614126 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4xxj\" (UniqueName: \"kubernetes.io/projected/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-kube-api-access-r4xxj\") pod \"maas-api-revoke-sub-8aa9e2f8-rsl6g\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.631058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.631026 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:24.642284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.642050 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:24.971850 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:09:24.969020 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bd54d1_91ab_4e96_a4cc_2ad4726018ba.slice/crio-fee01d9f6e0ee0f10bb3f3a9e6e57a0f9d9d34ad8f64d4a497013680c9ed70f2 WatchSource:0}: Error finding container fee01d9f6e0ee0f10bb3f3a9e6e57a0f9d9d34ad8f64d4a497013680c9ed70f2: Status 404 returned error can't find the container with id fee01d9f6e0ee0f10bb3f3a9e6e57a0f9d9d34ad8f64d4a497013680c9ed70f2 Oct 30 13:09:24.972836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.972802 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s"] Oct 30 13:09:24.973260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.972857 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.973260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.972938 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.973260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:24.972981 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:25.022783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.022701 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g"] Oct 30 13:09:25.034540 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:09:25.034508 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27f0e1a_01a0_4631_9d25_e058b3f5a90c.slice/crio-05eeaf9f53a64719de91cc065409439dae628a5cc4d83e46f5a84f9c21df29ed WatchSource:0}: Error finding container 05eeaf9f53a64719de91cc065409439dae628a5cc4d83e46f5a84f9c21df29ed: Status 404 returned error can't find the container with id 05eeaf9f53a64719de91cc065409439dae628a5cc4d83e46f5a84f9c21df29ed Oct 30 13:09:25.060771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.060740 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr"] Oct 30 13:09:25.062631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.060986 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:25.062631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.062237 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:25.062631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.062296 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:25.134212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.134161 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" event={"ID":"54b70acc-a40d-4065-8c58-53a6fff7d969","Type":"ContainerStarted","Data":"673bbe0b98019732a3c4a3a6d4f99e00d1e0a1ad9469397dc5d32e9346f39592"} Oct 30 13:09:25.145248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.144616 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" event={"ID":"86bd54d1-91ab-4e96-a4cc-2ad4726018ba","Type":"ContainerStarted","Data":"71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4"} Oct 30 13:09:25.145248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.144697 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" event={"ID":"86bd54d1-91ab-4e96-a4cc-2ad4726018ba","Type":"ContainerStarted","Data":"fee01d9f6e0ee0f10bb3f3a9e6e57a0f9d9d34ad8f64d4a497013680c9ed70f2"} Oct 30 13:09:25.148861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.148831 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" event={"ID":"d27f0e1a-01a0-4631-9d25-e058b3f5a90c","Type":"ContainerStarted","Data":"05eeaf9f53a64719de91cc065409439dae628a5cc4d83e46f5a84f9c21df29ed"} Oct 30 13:09:25.199372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.198438 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" podStartSLOduration=2.198420637 podStartE2EDuration="2.198420637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:25.196677359 +0000 UTC m=+1072.011871784" watchObservedRunningTime="2026-10-30 13:09:25.198420637 +0000 UTC m=+1072.013615062" Oct 30 13:09:25.238204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.238165 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:25.605964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:25.605931 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:26.183298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:26.183265 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" event={"ID":"d27f0e1a-01a0-4631-9d25-e058b3f5a90c","Type":"ContainerStarted","Data":"9f644e07e710a71963cd228f51c478a1698f3174f2b55970c15c1031fd3575bf"} Oct 30 13:09:26.186020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:26.185995 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" event={"ID":"54b70acc-a40d-4065-8c58-53a6fff7d969","Type":"ContainerStarted","Data":"4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88"} Oct 30 13:09:26.245466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:26.237774 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:26.329545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:26.321842 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" podStartSLOduration=2.321820786 podStartE2EDuration="2.321820786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:26.263300262 +0000 UTC m=+1073.078494688" watchObservedRunningTime="2026-10-30 13:09:26.321820786 +0000 UTC m=+1073.137015210" Oct 30 13:09:26.364346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:26.352179 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" podStartSLOduration=2.3521428699999998 podStartE2EDuration="2.35214287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:26.338207641 +0000 UTC m=+1073.153402065" watchObservedRunningTime="2026-10-30 13:09:26.35214287 +0000 UTC m=+1073.167337296" Oct 30 13:09:26.620719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:26.620678 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:09:26.621250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:26.620945 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 13:09:26.649304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:26.649271 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:26.649667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:26.649644 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 13:09:26.649808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:26.649787 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 13:09:26.692501 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:09:26.692465 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:09:27.200982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:27.199870 5049 generic.go:356] "Generic (PLEG): container finished" podID="d27f0e1a-01a0-4631-9d25-e058b3f5a90c" containerID="9f644e07e710a71963cd228f51c478a1698f3174f2b55970c15c1031fd3575bf" exitCode=0 Oct 30 13:09:27.200982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:27.200528 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" event={"ID":"d27f0e1a-01a0-4631-9d25-e058b3f5a90c","Type":"ContainerDied","Data":"9f644e07e710a71963cd228f51c478a1698f3174f2b55970c15c1031fd3575bf"} Oct 30 13:09:27.609861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:27.609821 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:09:27.610178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:27.610133 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 13:09:27.652354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:27.649912 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 13:09:27.652354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:27.649988 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 13:09:28.639372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.630733 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:09:28.639372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.630811 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 13:09:28.660973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.660847 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 13:09:28.660973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.660904 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 13:09:28.669857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.669465 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:28.758373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.743030 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-xks-ca\") pod \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " Oct 30 13:09:28.758373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.743097 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-openshift-ca\") pod \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " Oct 30 13:09:28.758373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.743182 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-kube-api-access-r4xxj\" (UniqueName: \"kubernetes.io/projected/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-kube-api-access-r4xxj\") pod \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " Oct 30 13:09:28.777353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.774705 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-openshift-ca" pod "d27f0e1a-01a0-4631-9d25-e058b3f5a90c" (UID: "d27f0e1a-01a0-4631-9d25-e058b3f5a90c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:28.881454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.876769 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:28.901004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.900902 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:29.001595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:28.990481 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-kube-api-access-r4xxj" pod "d27f0e1a-01a0-4631-9d25-e058b3f5a90c" (UID: "d27f0e1a-01a0-4631-9d25-e058b3f5a90c"). InnerVolumeSpecName "kube-api-access-r4xxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:29.085972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.085907 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-xks-ca" pod "d27f0e1a-01a0-4631-9d25-e058b3f5a90c" (UID: "d27f0e1a-01a0-4631-9d25-e058b3f5a90c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:29.086134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.086118 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-xks-ca\") pod \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\" (UID: \"d27f0e1a-01a0-4631-9d25-e058b3f5a90c\") " Oct 30 13:09:29.086845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.086804 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4xxj\" (UniqueName: \"kubernetes.io/projected/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-kube-api-access-r4xxj\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:29.087126 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:09:29.087099 5049 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/d27f0e1a-01a0-4631-9d25-e058b3f5a90c/volumes/kubernetes.io~secret/maas-api-xks-ca Oct 30 13:09:29.087254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.087237 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-xks-ca" pod "d27f0e1a-01a0-4631-9d25-e058b3f5a90c" (UID: "d27f0e1a-01a0-4631-9d25-e058b3f5a90c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:29.189078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.189001 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27f0e1a-01a0-4631-9d25-e058b3f5a90c-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:29.296335 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.296244 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" Oct 30 13:09:29.296524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.296243 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g" event={"ID":"d27f0e1a-01a0-4631-9d25-e058b3f5a90c","Type":"ContainerDied","Data":"05eeaf9f53a64719de91cc065409439dae628a5cc4d83e46f5a84f9c21df29ed"} Oct 30 13:09:29.296524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.296389 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05eeaf9f53a64719de91cc065409439dae628a5cc4d83e46f5a84f9c21df29ed" Oct 30 13:09:29.633346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.633297 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:09:29.633668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.633387 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 13:09:29.654346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.647704 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 13:09:29.654346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.647767 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 13:09:29.779721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.779677 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8"] Oct 30 13:09:29.780427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.780313 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d27f0e1a-01a0-4631-9d25-e058b3f5a90c" containerName="revoke-keys" Oct 30 13:09:29.780427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.780347 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27f0e1a-01a0-4631-9d25-e058b3f5a90c" containerName="revoke-keys" Oct 30 13:09:29.780678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.780658 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27f0e1a-01a0-4631-9d25-e058b3f5a90c" containerName="revoke-keys" Oct 30 13:09:29.790746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.790715 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8"] Oct 30 13:09:29.790877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.790831 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:29.811717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.811691 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:09:29.812019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.812001 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:09:29.901533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.901155 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeaa75ec-7h9r8\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:29.901533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.901197 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeaa75ec-7h9r8\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:29.901533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:29.901358 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxqqd\" (UniqueName: \"kubernetes.io/projected/143a1d42-5914-4774-8dc2-50d83050efe0-kube-api-access-lxqqd\") pod \"maas-api-revoke-sub-eeaa75ec-7h9r8\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:30.002561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.002529 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxqqd\" (UniqueName: \"kubernetes.io/projected/143a1d42-5914-4774-8dc2-50d83050efe0-kube-api-access-lxqqd\") pod \"maas-api-revoke-sub-eeaa75ec-7h9r8\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:30.002838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.002819 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeaa75ec-7h9r8\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:30.002951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.002937 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeaa75ec-7h9r8\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:30.003607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.003588 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeaa75ec-7h9r8\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:30.055346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.036883 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeaa75ec-7h9r8\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:30.055346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.045105 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxqqd\" (UniqueName: \"kubernetes.io/projected/143a1d42-5914-4774-8dc2-50d83050efe0-kube-api-access-lxqqd\") pod \"maas-api-revoke-sub-eeaa75ec-7h9r8\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:30.167399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.166745 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:30.583874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.583824 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8"] Oct 30 13:09:30.630373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.627959 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:09:30.630373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.628034 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 13:09:30.648363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.645083 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 13:09:30.648363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:30.645140 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 13:09:31.124438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:31.124394 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:31.354393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:31.352490 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" event={"ID":"143a1d42-5914-4774-8dc2-50d83050efe0","Type":"ContainerStarted","Data":"9cd1f88a46e445c22366d437d92b0e56aefa5e8c094a52f659c5e6fcb78a03a0"} Oct 30 13:09:31.354393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:31.352541 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" event={"ID":"143a1d42-5914-4774-8dc2-50d83050efe0","Type":"ContainerStarted","Data":"47199eb1ac58b9b2376333bab735b8cdf934dc8fefbe5829bbb6fc64cffda7e3"} Oct 30 13:09:31.399936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:31.399818 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" podStartSLOduration=2.399803153 podStartE2EDuration="2.399803153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:31.394382895 +0000 UTC m=+1078.209577318" watchObservedRunningTime="2026-10-30 13:09:31.399803153 +0000 UTC m=+1078.214997575" Oct 30 13:09:31.609347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:31.607759 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:09:31.609347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:31.607823 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 13:09:31.643338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:31.643291 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 13:09:31.643543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:31.643416 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 13:09:32.365122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.365083 5049 generic.go:356] "Generic (PLEG): container finished" podID="143a1d42-5914-4774-8dc2-50d83050efe0" containerID="9cd1f88a46e445c22366d437d92b0e56aefa5e8c094a52f659c5e6fcb78a03a0" exitCode=0 Oct 30 13:09:32.365627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.365165 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" event={"ID":"143a1d42-5914-4774-8dc2-50d83050efe0","Type":"ContainerDied","Data":"9cd1f88a46e445c22366d437d92b0e56aefa5e8c094a52f659c5e6fcb78a03a0"} Oct 30 13:09:32.536256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.536217 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n"] Oct 30 13:09:32.540764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.540746 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:32.568621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.568589 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n"] Oct 30 13:09:32.617309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.617227 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:32.619881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.619812 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:32.620017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.618243 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4031218c-nm46n\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:32.620283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.620267 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvlnw\" (UniqueName: \"kubernetes.io/projected/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-kube-api-access-rvlnw\") pod \"maas-api-revoke-sub-4031218c-nm46n\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:32.620547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.620529 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4031218c-nm46n\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:32.626352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.626006 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:09:32.722196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.722156 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4031218c-nm46n\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:32.722443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.722424 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvlnw\" (UniqueName: \"kubernetes.io/projected/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-kube-api-access-rvlnw\") pod \"maas-api-revoke-sub-4031218c-nm46n\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:32.722552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.722537 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4031218c-nm46n\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:32.724921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.723871 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4031218c-nm46n\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:32.727411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.727385 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4031218c-nm46n\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:32.793114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.793071 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvlnw\" (UniqueName: \"kubernetes.io/projected/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-kube-api-access-rvlnw\") pod \"maas-api-revoke-sub-4031218c-nm46n\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:32.842629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.840344 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:32.847267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.845693 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:32.854819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.854794 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:09:32.856072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:32.856051 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:33.183691 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:09:33.183660 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ce8cc0_d9e8_4415_8f5e_c210be33101f.slice/crio-9c30ed552c902e5e8117582aab3b9623200c696313a378281d9bdcd962f0550a WatchSource:0}: Error finding container 9c30ed552c902e5e8117582aab3b9623200c696313a378281d9bdcd962f0550a: Status 404 returned error can't find the container with id 9c30ed552c902e5e8117582aab3b9623200c696313a378281d9bdcd962f0550a Oct 30 13:09:33.184957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.184929 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n"] Oct 30 13:09:33.337692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.337645 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:33.377482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.377442 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" event={"ID":"e0ce8cc0-d9e8-4415-8f5e-c210be33101f","Type":"ContainerStarted","Data":"9c30ed552c902e5e8117582aab3b9623200c696313a378281d9bdcd962f0550a"} Oct 30 13:09:33.618016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.617991 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:33.641129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.641102 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143a1d42-5914-4774-8dc2-50d83050efe0-kube-api-access-lxqqd\" (UniqueName: \"kubernetes.io/projected/143a1d42-5914-4774-8dc2-50d83050efe0-kube-api-access-lxqqd\") pod \"143a1d42-5914-4774-8dc2-50d83050efe0\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " Oct 30 13:09:33.641242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.641148 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-xks-ca\") pod \"143a1d42-5914-4774-8dc2-50d83050efe0\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " Oct 30 13:09:33.641242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.641233 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-openshift-ca\") pod \"143a1d42-5914-4774-8dc2-50d83050efe0\" (UID: \"143a1d42-5914-4774-8dc2-50d83050efe0\") " Oct 30 13:09:33.641637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.641600 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-openshift-ca" pod "143a1d42-5914-4774-8dc2-50d83050efe0" (UID: "143a1d42-5914-4774-8dc2-50d83050efe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:33.641896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.641877 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:33.643185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.643162 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-xks-ca" pod "143a1d42-5914-4774-8dc2-50d83050efe0" (UID: "143a1d42-5914-4774-8dc2-50d83050efe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:33.643260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.643243 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143a1d42-5914-4774-8dc2-50d83050efe0-kube-api-access-lxqqd" pod "143a1d42-5914-4774-8dc2-50d83050efe0" (UID: "143a1d42-5914-4774-8dc2-50d83050efe0"). InnerVolumeSpecName "kube-api-access-lxqqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:33.743054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.743026 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxqqd\" (UniqueName: \"kubernetes.io/projected/143a1d42-5914-4774-8dc2-50d83050efe0-kube-api-access-lxqqd\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:33.743054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:33.743053 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143a1d42-5914-4774-8dc2-50d83050efe0-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:34.386743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:34.386714 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" Oct 30 13:09:34.386743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:34.386734 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8" event={"ID":"143a1d42-5914-4774-8dc2-50d83050efe0","Type":"ContainerDied","Data":"47199eb1ac58b9b2376333bab735b8cdf934dc8fefbe5829bbb6fc64cffda7e3"} Oct 30 13:09:34.387195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:34.386788 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47199eb1ac58b9b2376333bab735b8cdf934dc8fefbe5829bbb6fc64cffda7e3" Oct 30 13:09:34.388618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:34.388594 5049 generic.go:356] "Generic (PLEG): container finished" podID="e0ce8cc0-d9e8-4415-8f5e-c210be33101f" containerID="bcda07c0777c88641fa74f3496730252702b091ebaad7f7990580af81f30f44d" exitCode=0 Oct 30 13:09:34.388761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:34.388677 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" event={"ID":"e0ce8cc0-d9e8-4415-8f5e-c210be33101f","Type":"ContainerDied","Data":"bcda07c0777c88641fa74f3496730252702b091ebaad7f7990580af81f30f44d"} Oct 30 13:09:35.133853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.133818 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:35.572314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.572284 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:35.669532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.669446 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-kube-api-access-rvlnw\" (UniqueName: \"kubernetes.io/projected/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-kube-api-access-rvlnw\") pod \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " Oct 30 13:09:35.669708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.669659 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-xks-ca\") pod \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " Oct 30 13:09:35.669788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.669772 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-openshift-ca\") pod \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\" (UID: \"e0ce8cc0-d9e8-4415-8f5e-c210be33101f\") " Oct 30 13:09:35.670261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.670209 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-openshift-ca" pod "e0ce8cc0-d9e8-4415-8f5e-c210be33101f" (UID: "e0ce8cc0-d9e8-4415-8f5e-c210be33101f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:35.671803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.671773 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-xks-ca" pod "e0ce8cc0-d9e8-4415-8f5e-c210be33101f" (UID: "e0ce8cc0-d9e8-4415-8f5e-c210be33101f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:35.672047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.672024 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-kube-api-access-rvlnw" pod "e0ce8cc0-d9e8-4415-8f5e-c210be33101f" (UID: "e0ce8cc0-d9e8-4415-8f5e-c210be33101f"). InnerVolumeSpecName "kube-api-access-rvlnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:35.771692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.771655 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:35.771692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.771693 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvlnw\" (UniqueName: \"kubernetes.io/projected/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-kube-api-access-rvlnw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:35.771898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:35.771708 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ce8cc0-d9e8-4415-8f5e-c210be33101f-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:36.140903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:36.140872 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:36.407468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:36.407314 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" event={"ID":"e0ce8cc0-d9e8-4415-8f5e-c210be33101f","Type":"ContainerDied","Data":"9c30ed552c902e5e8117582aab3b9623200c696313a378281d9bdcd962f0550a"} Oct 30 13:09:36.407468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:36.407417 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c30ed552c902e5e8117582aab3b9623200c696313a378281d9bdcd962f0550a" Oct 30 13:09:36.407745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:36.407490 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n" Oct 30 13:09:37.106583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:37.104623 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:39.072430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:39.072372 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:40.189844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.189811 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs"] Oct 30 13:09:40.190519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.190498 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e0ce8cc0-d9e8-4415-8f5e-c210be33101f" containerName="revoke-keys" Oct 30 13:09:40.190519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.190517 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ce8cc0-d9e8-4415-8f5e-c210be33101f" containerName="revoke-keys" Oct 30 13:09:40.190735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.190533 5049 cpu_manager.go:414] "containerMap: removing container" podUID="143a1d42-5914-4774-8dc2-50d83050efe0" containerName="revoke-keys" Oct 30 13:09:40.190735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.190539 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143a1d42-5914-4774-8dc2-50d83050efe0" containerName="revoke-keys" Oct 30 13:09:40.190735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.190669 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ce8cc0-d9e8-4415-8f5e-c210be33101f" containerName="revoke-keys" Oct 30 13:09:40.190735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.190680 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="143a1d42-5914-4774-8dc2-50d83050efe0" containerName="revoke-keys" Oct 30 13:09:40.195784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.195767 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.218352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.218301 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs"] Oct 30 13:09:40.221893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.221861 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:09:40.222018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.221992 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:09:40.247284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.247247 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a2ff16c-72nqs\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.247687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.247661 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5468\" (UniqueName: \"kubernetes.io/projected/5ad1c025-580c-4307-a704-54b35c4f2bce-kube-api-access-h5468\") pod \"maas-api-revoke-sub-4a2ff16c-72nqs\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.247876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.247857 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a2ff16c-72nqs\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.349706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.349669 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a2ff16c-72nqs\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.349887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.349794 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5468\" (UniqueName: \"kubernetes.io/projected/5ad1c025-580c-4307-a704-54b35c4f2bce-kube-api-access-h5468\") pod \"maas-api-revoke-sub-4a2ff16c-72nqs\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.349961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.349947 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a2ff16c-72nqs\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.350616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.350588 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a2ff16c-72nqs\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.351991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.351966 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a2ff16c-72nqs\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.373645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.373617 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5468\" (UniqueName: \"kubernetes.io/projected/5ad1c025-580c-4307-a704-54b35c4f2bce-kube-api-access-h5468\") pod \"maas-api-revoke-sub-4a2ff16c-72nqs\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.523669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.523576 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:40.801460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.799639 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:40.867415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:40.867380 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs"] Oct 30 13:09:41.453898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:41.453804 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" event={"ID":"5ad1c025-580c-4307-a704-54b35c4f2bce","Type":"ContainerStarted","Data":"61968ad398a1af1845276a8be2335f92bc8a246dfa0c1d1e449c3d684c012f95"} Oct 30 13:09:41.453898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:41.453855 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" event={"ID":"5ad1c025-580c-4307-a704-54b35c4f2bce","Type":"ContainerStarted","Data":"4ff2546f0cea33f2542cbfd2b5b1b343cc537f7858f2e007fd47fa1a83d615f5"} Oct 30 13:09:41.495959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:41.495905 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" podStartSLOduration=1.495890849 podStartE2EDuration="1.495890849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:41.494086353 +0000 UTC m=+1088.309280780" watchObservedRunningTime="2026-10-30 13:09:41.495890849 +0000 UTC m=+1088.311085287" Oct 30 13:09:42.465254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:42.465212 5049 generic.go:356] "Generic (PLEG): container finished" podID="5ad1c025-580c-4307-a704-54b35c4f2bce" containerID="61968ad398a1af1845276a8be2335f92bc8a246dfa0c1d1e449c3d684c012f95" exitCode=0 Oct 30 13:09:42.465627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:42.465357 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" event={"ID":"5ad1c025-580c-4307-a704-54b35c4f2bce","Type":"ContainerDied","Data":"61968ad398a1af1845276a8be2335f92bc8a246dfa0c1d1e449c3d684c012f95"} Oct 30 13:09:42.677552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:42.677507 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:42.903629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:42.903594 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt"] Oct 30 13:09:42.908498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:42.908475 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:42.931280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:42.931252 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt"] Oct 30 13:09:42.989123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:42.989082 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d776433-l25qt\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:42.989286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:42.989156 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d776433-l25qt\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:42.989350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:42.989310 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wmzz\" (UniqueName: \"kubernetes.io/projected/799ff105-9b86-46ca-89f7-a581e94e6100-kube-api-access-4wmzz\") pod \"maas-api-revoke-sub-4d776433-l25qt\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:43.090455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.090411 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d776433-l25qt\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:43.090655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.090493 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d776433-l25qt\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:43.090655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.090633 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmzz\" (UniqueName: \"kubernetes.io/projected/799ff105-9b86-46ca-89f7-a581e94e6100-kube-api-access-4wmzz\") pod \"maas-api-revoke-sub-4d776433-l25qt\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:43.091195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.091170 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d776433-l25qt\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:43.092778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.092759 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d776433-l25qt\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:43.115043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.115015 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmzz\" (UniqueName: \"kubernetes.io/projected/799ff105-9b86-46ca-89f7-a581e94e6100-kube-api-access-4wmzz\") pod \"maas-api-revoke-sub-4d776433-l25qt\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:43.226647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.226565 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:43.568367 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:09:43.560756 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799ff105_9b86_46ca_89f7_a581e94e6100.slice/crio-47cbe11097f5ed210d7993d5c99568fb83781a6d33c183c8797927e3946fe315 WatchSource:0}: Error finding container 47cbe11097f5ed210d7993d5c99568fb83781a6d33c183c8797927e3946fe315: Status 404 returned error can't find the container with id 47cbe11097f5ed210d7993d5c99568fb83781a6d33c183c8797927e3946fe315 Oct 30 13:09:43.568367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.567565 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt"] Oct 30 13:09:43.636678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.636649 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:43.697035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.697001 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad1c025-580c-4307-a704-54b35c4f2bce-kube-api-access-h5468\" (UniqueName: \"kubernetes.io/projected/5ad1c025-580c-4307-a704-54b35c4f2bce-kube-api-access-h5468\") pod \"5ad1c025-580c-4307-a704-54b35c4f2bce\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " Oct 30 13:09:43.697218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.697196 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-xks-ca\") pod \"5ad1c025-580c-4307-a704-54b35c4f2bce\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " Oct 30 13:09:43.697291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.697235 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-openshift-ca\") pod \"5ad1c025-580c-4307-a704-54b35c4f2bce\" (UID: \"5ad1c025-580c-4307-a704-54b35c4f2bce\") " Oct 30 13:09:43.697707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.697678 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-openshift-ca" pod "5ad1c025-580c-4307-a704-54b35c4f2bce" (UID: "5ad1c025-580c-4307-a704-54b35c4f2bce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:43.697807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.697768 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:43.699526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.699495 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-xks-ca" pod "5ad1c025-580c-4307-a704-54b35c4f2bce" (UID: "5ad1c025-580c-4307-a704-54b35c4f2bce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:43.699741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.699719 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad1c025-580c-4307-a704-54b35c4f2bce-kube-api-access-h5468" pod "5ad1c025-580c-4307-a704-54b35c4f2bce" (UID: "5ad1c025-580c-4307-a704-54b35c4f2bce"). InnerVolumeSpecName "kube-api-access-h5468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:43.799806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.799772 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad1c025-580c-4307-a704-54b35c4f2bce-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:43.799806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:43.799801 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5468\" (UniqueName: \"kubernetes.io/projected/5ad1c025-580c-4307-a704-54b35c4f2bce-kube-api-access-h5468\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:44.233060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:44.233027 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:44.488386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:44.488270 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" event={"ID":"799ff105-9b86-46ca-89f7-a581e94e6100","Type":"ContainerStarted","Data":"3e29155a3ca1230f415d89359022452c0dc351a996f8e4c06bacba661121ee20"} Oct 30 13:09:44.488386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:44.488373 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" event={"ID":"799ff105-9b86-46ca-89f7-a581e94e6100","Type":"ContainerStarted","Data":"47cbe11097f5ed210d7993d5c99568fb83781a6d33c183c8797927e3946fe315"} Oct 30 13:09:44.489811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:44.489782 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" Oct 30 13:09:44.489935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:44.489818 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs" event={"ID":"5ad1c025-580c-4307-a704-54b35c4f2bce","Type":"ContainerDied","Data":"4ff2546f0cea33f2542cbfd2b5b1b343cc537f7858f2e007fd47fa1a83d615f5"} Oct 30 13:09:44.489935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:44.489862 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ff2546f0cea33f2542cbfd2b5b1b343cc537f7858f2e007fd47fa1a83d615f5" Oct 30 13:09:44.529937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:44.529861 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" podStartSLOduration=2.529840072 podStartE2EDuration="2.529840072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:44.528297715 +0000 UTC m=+1091.343492143" watchObservedRunningTime="2026-10-30 13:09:44.529840072 +0000 UTC m=+1091.345034496" Oct 30 13:09:44.745580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:44.745481 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:45.501425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:45.501395 5049 generic.go:356] "Generic (PLEG): container finished" podID="799ff105-9b86-46ca-89f7-a581e94e6100" containerID="3e29155a3ca1230f415d89359022452c0dc351a996f8e4c06bacba661121ee20" exitCode=0 Oct 30 13:09:45.501581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:45.501476 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" event={"ID":"799ff105-9b86-46ca-89f7-a581e94e6100","Type":"ContainerDied","Data":"3e29155a3ca1230f415d89359022452c0dc351a996f8e4c06bacba661121ee20"} Oct 30 13:09:46.472728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:46.472694 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:46.654212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:46.654171 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:46.872961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:46.872914 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:46.958742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:46.958705 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-openshift-ca\") pod \"799ff105-9b86-46ca-89f7-a581e94e6100\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " Oct 30 13:09:46.958915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:46.958758 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-xks-ca\") pod \"799ff105-9b86-46ca-89f7-a581e94e6100\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " Oct 30 13:09:46.958915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:46.958787 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799ff105-9b86-46ca-89f7-a581e94e6100-kube-api-access-4wmzz\" (UniqueName: \"kubernetes.io/projected/799ff105-9b86-46ca-89f7-a581e94e6100-kube-api-access-4wmzz\") pod \"799ff105-9b86-46ca-89f7-a581e94e6100\" (UID: \"799ff105-9b86-46ca-89f7-a581e94e6100\") " Oct 30 13:09:46.959181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:46.959144 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-openshift-ca" pod "799ff105-9b86-46ca-89f7-a581e94e6100" (UID: "799ff105-9b86-46ca-89f7-a581e94e6100"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:46.959408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:46.959389 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:46.961229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:46.961200 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799ff105-9b86-46ca-89f7-a581e94e6100-kube-api-access-4wmzz" pod "799ff105-9b86-46ca-89f7-a581e94e6100" (UID: "799ff105-9b86-46ca-89f7-a581e94e6100"). InnerVolumeSpecName "kube-api-access-4wmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:46.961481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:46.961457 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-xks-ca" pod "799ff105-9b86-46ca-89f7-a581e94e6100" (UID: "799ff105-9b86-46ca-89f7-a581e94e6100"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:47.061075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:47.060983 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799ff105-9b86-46ca-89f7-a581e94e6100-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:47.061075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:47.061020 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wmzz\" (UniqueName: \"kubernetes.io/projected/799ff105-9b86-46ca-89f7-a581e94e6100-kube-api-access-4wmzz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:09:47.277225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:47.277189 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:47.527193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:47.527156 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" event={"ID":"799ff105-9b86-46ca-89f7-a581e94e6100","Type":"ContainerDied","Data":"47cbe11097f5ed210d7993d5c99568fb83781a6d33c183c8797927e3946fe315"} Oct 30 13:09:47.527610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:47.527224 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47cbe11097f5ed210d7993d5c99568fb83781a6d33c183c8797927e3946fe315" Oct 30 13:09:47.527610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:47.527170 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt" Oct 30 13:09:48.617836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:48.617800 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:50.376393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:50.369412 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.576466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:50.576419 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.783524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:50.783419 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.801472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:50.801423 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.822491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:50.822409 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.099994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.093433 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.316347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.249395 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:09:51.316347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.249447 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.316347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.275535 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:09:51.346343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.342443 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.374291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.373404 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.786240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.786144 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.802941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.802906 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:09:51.809453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.809425 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:09:51.838254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.838197 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:09:51.840038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:51.840000 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:09:52.174604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.173871 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.179126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.175347 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:09:52.270553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.261468 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:09:52.290354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.272652 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:09:52.376421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.273609 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.378967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.290134 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.402365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.380525 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.402365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.380661 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.527401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.523363 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:09:52.527401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.523822 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:09:52.527401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.523975 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:09:52.861352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.857380 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:09:52.885712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.885673 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:09:52.995526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:52.995472 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 13:09:53.175395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:53.174517 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:53.372815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:53.372773 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:53.616442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:53.415387 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:09:53.617087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:53.417492 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:53.621654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:53.417522 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:53.622171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:53.452403 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:53.658909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:53.658864 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:53.874468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:53.874345 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:09:53.916523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:53.916463 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:54.015352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:53.997878 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:54.034355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.028793 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:09:54.038380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.035423 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:54.054059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.053657 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:54.132628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.132543 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:09:54.184455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.184412 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:54.198717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.198677 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:09:54.557400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.521552 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:54.600645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.593109 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:09:54.643546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.622552 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:54.669922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.669877 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:09:54.783726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.783692 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:54.906415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:54.897919 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:09:55.007981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.007949 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n"] Oct 30 13:09:55.017188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.017158 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5ad1c025-580c-4307-a704-54b35c4f2bce" containerName="revoke-keys" Oct 30 13:09:55.017846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.017820 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad1c025-580c-4307-a704-54b35c4f2bce" containerName="revoke-keys" Oct 30 13:09:55.018115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.018099 5049 cpu_manager.go:414] "containerMap: removing container" podUID="799ff105-9b86-46ca-89f7-a581e94e6100" containerName="revoke-keys" Oct 30 13:09:55.050023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.049993 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799ff105-9b86-46ca-89f7-a581e94e6100" containerName="revoke-keys" Oct 30 13:09:55.050845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.050816 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="799ff105-9b86-46ca-89f7-a581e94e6100" containerName="revoke-keys" Oct 30 13:09:55.051020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.051006 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad1c025-580c-4307-a704-54b35c4f2bce" containerName="revoke-keys" Oct 30 13:09:55.065495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.065452 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:55.073941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.036729 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 13:09:55.082733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.048820 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:55.084902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.084757 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b"] Oct 30 13:09:55.092154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.092129 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.106428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.105471 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.106428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.105548 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.106428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.105623 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55408942-f891-43f8-b85f-e5c4345c23fd-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.106428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.105685 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.106428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.105776 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.106428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.105856 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55408942-f891-43f8-b85f-e5c4345c23fd-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.106428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.105899 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.106428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.105947 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.106428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.106035 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gprh\" (UniqueName: \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-kube-api-access-9gprh\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.121055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.121015 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:09:55.121345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.121328 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:09:55.121503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.121490 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n"] Oct 30 13:09:55.122908 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.122887 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.139973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.139893 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b"] Oct 30 13:09:55.173857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.173827 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:09:55.177057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.177033 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.227220 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55408942-f891-43f8-b85f-e5c4345c23fd-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.243105 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.243230 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.243288 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55408942-f891-43f8-b85f-e5c4345c23fd-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.241832 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55408942-f891-43f8-b85f-e5c4345c23fd-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.244371 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.244429 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.244558 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gprh\" (UniqueName: \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-kube-api-access-9gprh\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.244665 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.244755 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.245069 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.250088 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.250621 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.256947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.251472 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.358359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.356811 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:55.369340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.368129 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.369340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.368364 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx6mz\" (UniqueName: \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-kube-api-access-wx6mz\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.369340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.368412 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.369340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.368442 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd548e-8a90-4a57-8046-c5731787e45a-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.369340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.368482 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.369340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.368541 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.369340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.368582 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd548e-8a90-4a57-8046-c5731787e45a-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.369340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.368629 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.369340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.368659 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.401671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.400764 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.427438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.415119 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.491001 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gprh\" (UniqueName: \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-kube-api-access-9gprh\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.491656 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55408942-f891-43f8-b85f-e5c4345c23fd-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.492960 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.493172 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx6mz\" (UniqueName: \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-kube-api-access-wx6mz\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.493225 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.493270 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd548e-8a90-4a57-8046-c5731787e45a-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.493313 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.493390 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.493433 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd548e-8a90-4a57-8046-c5731787e45a-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.493493 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.493526 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.493978 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.500110 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.501400 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.502021 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd548e-8a90-4a57-8046-c5731787e45a-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.504716 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.505760 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd548e-8a90-4a57-8046-c5731787e45a-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.511112 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.530427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.526474 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:55.540213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.539744 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx6mz\" (UniqueName: \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-kube-api-access-wx6mz\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.541234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.540425 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.657985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.654701 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:55.695521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.692066 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:55.954627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.949691 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:55.988567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.988521 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:55.988761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.988615 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:55.988761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.988660 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:55.991111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:55.991087 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n"] Oct 30 13:09:56.120776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.120694 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:56.158616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.158586 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b"] Oct 30 13:09:56.201343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.193374 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:56.201343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.193944 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:56.201343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.194029 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:56.291447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.291414 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" event={"ID":"55408942-f891-43f8-b85f-e5c4345c23fd","Type":"ContainerStarted","Data":"3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738"} Oct 30 13:09:56.291707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.291661 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" event={"ID":"55408942-f891-43f8-b85f-e5c4345c23fd","Type":"ContainerStarted","Data":"e73f00ca5912de6542e0f070303867c6815f712eb318d34c5bc8271bd9b94a1f"} Oct 30 13:09:56.298926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.298853 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" event={"ID":"dafd548e-8a90-4a57-8046-c5731787e45a","Type":"ContainerStarted","Data":"c7fbc05d900df352ac19ced67dd660772f726065e0a6bc43a428fdaf618b0136"} Oct 30 13:09:56.337260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.331086 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:56.387510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.384095 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" podStartSLOduration=4.384068954 podStartE2EDuration="4.384068954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:56.383687942 +0000 UTC m=+1103.198882366" watchObservedRunningTime="2026-10-30 13:09:56.384068954 +0000 UTC m=+1103.199263377" Oct 30 13:09:56.516783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.516690 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:56.534578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.534483 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:09:56.710532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.696407 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s"] Oct 30 13:09:56.712419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.712265 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:56.712862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.712814 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s"] Oct 30 13:09:56.713344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.713308 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:56.736179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.735605 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:09:56.769712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.769636 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:09:56.807352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.806750 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts7j5\" (UniqueName: \"kubernetes.io/projected/0d62dff5-a62d-4b68-bd98-52388bb76f92-kube-api-access-ts7j5\") pod \"maas-api-revoke-sub-310a6533-tw57s\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:56.807352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.806943 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-310a6533-tw57s\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:56.807352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.807042 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-310a6533-tw57s\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:56.814069 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:09:56.814032 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:09:56.833124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.833093 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v"] Oct 30 13:09:56.839034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.839000 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:56.859119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.859089 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7e2137bc-gw-openshift-default-dockercfg-bjmkl\"" Oct 30 13:09:56.875384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.875354 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v"] Oct 30 13:09:56.928350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.928258 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts7j5\" (UniqueName: \"kubernetes.io/projected/0d62dff5-a62d-4b68-bd98-52388bb76f92-kube-api-access-ts7j5\") pod \"maas-api-revoke-sub-310a6533-tw57s\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:56.928350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.928339 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-token\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:56.936571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.936537 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-envoy\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:56.936910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.936885 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-credential-socket\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:56.937079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.937061 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-socket\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:56.937251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.937233 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-310a6533-tw57s\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:56.937536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.937500 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-310a6533-tw57s\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:56.937720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.937703 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-podinfo\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:56.937876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.937857 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-certs\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:56.938044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.938028 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vq8x\" (UniqueName: \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-kube-api-access-5vq8x\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:56.938170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.938153 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53322eb2-900b-48e6-8be4-fb7d1a655f22-istiod-ca-cert\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:56.938272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.938258 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-data\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:56.942297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.942271 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-310a6533-tw57s\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:56.984457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.984426 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts7j5\" (UniqueName: \"kubernetes.io/projected/0d62dff5-a62d-4b68-bd98-52388bb76f92-kube-api-access-ts7j5\") pod \"maas-api-revoke-sub-310a6533-tw57s\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:57.005802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:56.996006 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-310a6533-tw57s\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:57.005802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.004435 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:57.067701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.067662 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-credential-socket\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.067938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.067919 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-socket\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.068213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.068196 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-podinfo\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.068380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.068364 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-certs\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.068564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.068546 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vq8x\" (UniqueName: \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-kube-api-access-5vq8x\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.085479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.085446 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53322eb2-900b-48e6-8be4-fb7d1a655f22-istiod-ca-cert\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.085701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.085683 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-data\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.086256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.086223 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-certs\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.086378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.086290 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-token\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.086459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.086389 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-envoy\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.092699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.092623 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53322eb2-900b-48e6-8be4-fb7d1a655f22-istiod-ca-cert\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.092866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.074756 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-socket\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.093036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.074985 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-credential-socket\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.096478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.096454 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:09:57.097648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.096494 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-data\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.099003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.098938 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-podinfo\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.101204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.101186 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-envoy\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.163351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.161741 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vq8x\" (UniqueName: \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-kube-api-access-5vq8x\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.181303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.176437 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-token\") pod \"e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.383402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.382809 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" event={"ID":"dafd548e-8a90-4a57-8046-c5731787e45a","Type":"ContainerStarted","Data":"198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6"} Oct 30 13:09:57.466756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.456708 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" podStartSLOduration=4.456686888 podStartE2EDuration="4.456686888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:57.438067493 +0000 UTC m=+1104.253261917" watchObservedRunningTime="2026-10-30 13:09:57.456686888 +0000 UTC m=+1104.271881311" Oct 30 13:09:57.466756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.461395 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:09:57.566122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.565535 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:09:57.566122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.565597 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 13:09:57.705472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.705434 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:09:57.711516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.711168 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 13:09:57.711516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.711229 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 13:09:57.897106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:57.897070 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:58.492531 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.491634 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs"] Oct 30 13:09:58.500281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.500244 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:58.500396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.500342 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:58.500396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.500386 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:58.604816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.531177 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:58.604816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.566364 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:09:58.604816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.566428 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 13:09:58.624366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.607383 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs"] Oct 30 13:09:58.728994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.727295 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 13:09:58.728994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.727369 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 13:09:58.777766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.755659 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:58.777766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.758553 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adb85511-ggbgs\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:58.777766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.758698 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggzh4\" (UniqueName: \"kubernetes.io/projected/80391d09-653e-4056-aa8d-05768237b1c8-kube-api-access-ggzh4\") pod \"maas-api-revoke-sub-adb85511-ggbgs\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:58.777766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.758866 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adb85511-ggbgs\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:58.782062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.781822 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s"] Oct 30 13:09:58.792409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.790208 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v"] Oct 30 13:09:58.883499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.879371 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adb85511-ggbgs\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:58.883499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.879532 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggzh4\" (UniqueName: \"kubernetes.io/projected/80391d09-653e-4056-aa8d-05768237b1c8-kube-api-access-ggzh4\") pod \"maas-api-revoke-sub-adb85511-ggbgs\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:58.883499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.879715 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adb85511-ggbgs\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:58.902438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.894232 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adb85511-ggbgs\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:58.909628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.909591 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adb85511-ggbgs\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:58.951851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:58.931437 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggzh4\" (UniqueName: \"kubernetes.io/projected/80391d09-653e-4056-aa8d-05768237b1c8-kube-api-access-ggzh4\") pod \"maas-api-revoke-sub-adb85511-ggbgs\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:59.093820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.093785 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:09:59.573422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.573381 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:09:59.586485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.586438 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 13:09:59.699985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.699900 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 13:09:59.699985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.699962 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 13:09:59.860657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.860610 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" event={"ID":"0d62dff5-a62d-4b68-bd98-52388bb76f92","Type":"ContainerStarted","Data":"711abda83face991f9902306e7b4f08d6332849ca113d9e32334937ccbbf13f7"} Oct 30 13:09:59.861811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.861778 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" event={"ID":"0d62dff5-a62d-4b68-bd98-52388bb76f92","Type":"ContainerStarted","Data":"053a17923a51ba7d20ee4b2a83495753bdaa64e61987f60a2c95fd78c4137320"} Oct 30 13:09:59.877371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.874998 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs"] Oct 30 13:09:59.877641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.877614 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" event={"ID":"53322eb2-900b-48e6-8be4-fb7d1a655f22","Type":"ContainerStarted","Data":"76ac6b5007e3e0c39bb1441c335addc6a4387fb361a2fc3b92b6912a9a7bc010"} Oct 30 13:09:59.877810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.877784 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" event={"ID":"53322eb2-900b-48e6-8be4-fb7d1a655f22","Type":"ContainerStarted","Data":"82a79f59b05e5fdf60ba21b3c327c2f28c7a5ff119c251b746191339727e0652"} Oct 30 13:09:59.985399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.985343 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" podStartSLOduration=3.985311352 podStartE2EDuration="3.985311352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:59.905359706 +0000 UTC m=+1106.720554132" watchObservedRunningTime="2026-10-30 13:09:59.985311352 +0000 UTC m=+1106.800505774" Oct 30 13:09:59.989804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:09:59.989762 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" podStartSLOduration=3.989746171 podStartE2EDuration="3.989746171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:59.978451066 +0000 UTC m=+1106.793645516" watchObservedRunningTime="2026-10-30 13:09:59.989746171 +0000 UTC m=+1106.804940594" Oct 30 13:10:00.324798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.324762 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:00.474560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.474385 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:10:00.479601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.479577 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:10:00.529606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.529520 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:10:00.529843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.529818 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 13:10:00.689622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.689585 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:00.693518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.693492 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 13:10:00.693649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.693539 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 13:10:00.889039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.889003 5049 generic.go:356] "Generic (PLEG): container finished" podID="0d62dff5-a62d-4b68-bd98-52388bb76f92" containerID="711abda83face991f9902306e7b4f08d6332849ca113d9e32334937ccbbf13f7" exitCode=0 Oct 30 13:10:00.889383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.889185 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" event={"ID":"0d62dff5-a62d-4b68-bd98-52388bb76f92","Type":"ContainerDied","Data":"711abda83face991f9902306e7b4f08d6332849ca113d9e32334937ccbbf13f7"} Oct 30 13:10:00.891415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.891386 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:00.892050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.892027 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" event={"ID":"80391d09-653e-4056-aa8d-05768237b1c8","Type":"ContainerStarted","Data":"89877fa87696d73c205fa5343cb25595e3f4c665c577d352f5e0aa0f39fbb5f2"} Oct 30 13:10:00.892141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.892075 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" event={"ID":"80391d09-653e-4056-aa8d-05768237b1c8","Type":"ContainerStarted","Data":"a9f51d9165a0c1bca2ad812c4085614a14ef0eda37061bdc75f9cc5536b48781"} Oct 30 13:10:00.892411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.892392 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:10:00.893285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:00.893264 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:10:01.092849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:01.092808 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:01.127343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:01.105715 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" podStartSLOduration=3.105695759 podStartE2EDuration="3.105695759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:01.093118454 +0000 UTC m=+1107.908312881" watchObservedRunningTime="2026-10-30 13:10:01.105695759 +0000 UTC m=+1107.920890188" Oct 30 13:10:01.531003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:01.530907 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:10:01.531260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:01.531233 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 13:10:01.693148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:01.693118 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 13:10:01.693625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:01.693600 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 13:10:01.902220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:01.902184 5049 generic.go:356] "Generic (PLEG): container finished" podID="80391d09-653e-4056-aa8d-05768237b1c8" containerID="89877fa87696d73c205fa5343cb25595e3f4c665c577d352f5e0aa0f39fbb5f2" exitCode=0 Oct 30 13:10:01.903124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:01.902605 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" event={"ID":"80391d09-653e-4056-aa8d-05768237b1c8","Type":"ContainerDied","Data":"89877fa87696d73c205fa5343cb25595e3f4c665c577d352f5e0aa0f39fbb5f2"} Oct 30 13:10:02.275460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.264427 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:02.376345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.375634 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:10:02.442345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.441972 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:02.513343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.502471 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d62dff5-a62d-4b68-bd98-52388bb76f92-kube-api-access-ts7j5\" (UniqueName: \"kubernetes.io/projected/0d62dff5-a62d-4b68-bd98-52388bb76f92-kube-api-access-ts7j5\") pod \"0d62dff5-a62d-4b68-bd98-52388bb76f92\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " Oct 30 13:10:02.513343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.502574 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-openshift-ca\") pod \"0d62dff5-a62d-4b68-bd98-52388bb76f92\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " Oct 30 13:10:02.513343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.502660 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-xks-ca\") pod \"0d62dff5-a62d-4b68-bd98-52388bb76f92\" (UID: \"0d62dff5-a62d-4b68-bd98-52388bb76f92\") " Oct 30 13:10:02.585949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.585557 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-openshift-ca" pod "0d62dff5-a62d-4b68-bd98-52388bb76f92" (UID: "0d62dff5-a62d-4b68-bd98-52388bb76f92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:02.795582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.607718 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:10:02.795582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.607783 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 13:10:02.795582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.717623 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:02.848850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.808640 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 13:10:02.848850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.808701 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 13:10:02.976345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.953013 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:02.977210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.966432 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" Oct 30 13:10:02.977511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.966624 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s" event={"ID":"0d62dff5-a62d-4b68-bd98-52388bb76f92","Type":"ContainerDied","Data":"053a17923a51ba7d20ee4b2a83495753bdaa64e61987f60a2c95fd78c4137320"} Oct 30 13:10:02.977699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.977682 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="053a17923a51ba7d20ee4b2a83495753bdaa64e61987f60a2c95fd78c4137320" Oct 30 13:10:02.983057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:02.983034 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:03.003752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.003709 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-xks-ca" pod "0d62dff5-a62d-4b68-bd98-52388bb76f92" (UID: "0d62dff5-a62d-4b68-bd98-52388bb76f92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:03.003919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.003780 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d62dff5-a62d-4b68-bd98-52388bb76f92-kube-api-access-ts7j5" pod "0d62dff5-a62d-4b68-bd98-52388bb76f92" (UID: "0d62dff5-a62d-4b68-bd98-52388bb76f92"). InnerVolumeSpecName "kube-api-access-ts7j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:03.012405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.012373 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt"] Oct 30 13:10:03.014957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.014933 5049 cpu_manager.go:414] "containerMap: removing container" podUID="0d62dff5-a62d-4b68-bd98-52388bb76f92" containerName="revoke-keys" Oct 30 13:10:03.015113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.015097 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d62dff5-a62d-4b68-bd98-52388bb76f92" containerName="revoke-keys" Oct 30 13:10:03.015567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.015539 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d62dff5-a62d-4b68-bd98-52388bb76f92" containerName="revoke-keys" Oct 30 13:10:03.079739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.079708 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:10:03.086583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.086508 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts7j5\" (UniqueName: \"kubernetes.io/projected/0d62dff5-a62d-4b68-bd98-52388bb76f92-kube-api-access-ts7j5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:03.086583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.086537 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d62dff5-a62d-4b68-bd98-52388bb76f92-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:03.090405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.090379 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt"] Oct 30 13:10:03.097272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.097245 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b31c2926-dockercfg-49blb\"" Oct 30 13:10:03.097272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.097252 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b31c2926\"" Oct 30 13:10:03.194011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.188973 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ffd100-3d77-4848-8001-72320ba194d4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt\" (UID: \"94ffd100-3d77-4848-8001-72320ba194d4\") " pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:10:03.194011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.189062 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw4fv\" (UniqueName: \"kubernetes.io/projected/94ffd100-3d77-4848-8001-72320ba194d4-kube-api-access-sw4fv\") pod \"payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt\" (UID: \"94ffd100-3d77-4848-8001-72320ba194d4\") " pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:10:03.300469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.294244 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:03.304339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.302190 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ffd100-3d77-4848-8001-72320ba194d4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt\" (UID: \"94ffd100-3d77-4848-8001-72320ba194d4\") " pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:10:03.304339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.302291 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw4fv\" (UniqueName: \"kubernetes.io/projected/94ffd100-3d77-4848-8001-72320ba194d4-kube-api-access-sw4fv\") pod \"payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt\" (UID: \"94ffd100-3d77-4848-8001-72320ba194d4\") " pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:10:03.304339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.304001 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ffd100-3d77-4848-8001-72320ba194d4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt\" (UID: \"94ffd100-3d77-4848-8001-72320ba194d4\") " pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:10:03.375098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.375018 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw4fv\" (UniqueName: \"kubernetes.io/projected/94ffd100-3d77-4848-8001-72320ba194d4-kube-api-access-sw4fv\") pod \"payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt\" (UID: \"94ffd100-3d77-4848-8001-72320ba194d4\") " pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:10:03.388836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.388791 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:10:03.398741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.398713 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:10:03.404902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.404874 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-openshift-ca\") pod \"80391d09-653e-4056-aa8d-05768237b1c8\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " Oct 30 13:10:03.411785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.411757 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-xks-ca\") pod \"80391d09-653e-4056-aa8d-05768237b1c8\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " Oct 30 13:10:03.417473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.417443 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80391d09-653e-4056-aa8d-05768237b1c8-kube-api-access-ggzh4\" (UniqueName: \"kubernetes.io/projected/80391d09-653e-4056-aa8d-05768237b1c8-kube-api-access-ggzh4\") pod \"80391d09-653e-4056-aa8d-05768237b1c8\" (UID: \"80391d09-653e-4056-aa8d-05768237b1c8\") " Oct 30 13:10:03.457686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.431589 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-openshift-ca" pod "80391d09-653e-4056-aa8d-05768237b1c8" (UID: "80391d09-653e-4056-aa8d-05768237b1c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:03.458727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.458698 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80391d09-653e-4056-aa8d-05768237b1c8-kube-api-access-ggzh4" pod "80391d09-653e-4056-aa8d-05768237b1c8" (UID: "80391d09-653e-4056-aa8d-05768237b1c8"). InnerVolumeSpecName "kube-api-access-ggzh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:03.460720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.460692 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-xks-ca" pod "80391d09-653e-4056-aa8d-05768237b1c8" (UID: "80391d09-653e-4056-aa8d-05768237b1c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:03.547059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.546979 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:03.547059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.547015 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggzh4\" (UniqueName: \"kubernetes.io/projected/80391d09-653e-4056-aa8d-05768237b1c8-kube-api-access-ggzh4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:03.547059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.547032 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80391d09-653e-4056-aa8d-05768237b1c8-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:03.547516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.547495 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:10:03.547726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.547701 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 13:10:03.693799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.693706 5049 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 13:10:03.693799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.693760 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 13:10:03.823510 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:03.823477 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ffd100_3d77_4848_8001_72320ba194d4.slice/crio-7d5009cd059df35ec031110f2d9a12d20f5ec86e3bc2cfd6c1cf2bcf3b45c7f9 WatchSource:0}: Error finding container 7d5009cd059df35ec031110f2d9a12d20f5ec86e3bc2cfd6c1cf2bcf3b45c7f9: Status 404 returned error can't find the container with id 7d5009cd059df35ec031110f2d9a12d20f5ec86e3bc2cfd6c1cf2bcf3b45c7f9 Oct 30 13:10:03.823977 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.823754 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt"] Oct 30 13:10:03.978480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.978390 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" Oct 30 13:10:03.978480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.978394 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs" event={"ID":"80391d09-653e-4056-aa8d-05768237b1c8","Type":"ContainerDied","Data":"a9f51d9165a0c1bca2ad812c4085614a14ef0eda37061bdc75f9cc5536b48781"} Oct 30 13:10:03.978919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.978500 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9f51d9165a0c1bca2ad812c4085614a14ef0eda37061bdc75f9cc5536b48781" Oct 30 13:10:03.981057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.981033 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" event={"ID":"94ffd100-3d77-4848-8001-72320ba194d4","Type":"ContainerStarted","Data":"5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4"} Oct 30 13:10:03.981179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:03.981093 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" event={"ID":"94ffd100-3d77-4848-8001-72320ba194d4","Type":"ContainerStarted","Data":"7d5009cd059df35ec031110f2d9a12d20f5ec86e3bc2cfd6c1cf2bcf3b45c7f9"} Oct 30 13:10:04.020155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.019944 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" podStartSLOduration=2.019931696 podStartE2EDuration="2.019931696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:04.017950029 +0000 UTC m=+1110.833144454" watchObservedRunningTime="2026-10-30 13:10:04.019931696 +0000 UTC m=+1110.835126116" Oct 30 13:10:04.389501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.389448 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:04.533887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.533847 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4"] Oct 30 13:10:04.534971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.534943 5049 cpu_manager.go:414] "containerMap: removing container" podUID="80391d09-653e-4056-aa8d-05768237b1c8" containerName="revoke-keys" Oct 30 13:10:04.534971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.534969 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80391d09-653e-4056-aa8d-05768237b1c8" containerName="revoke-keys" Oct 30 13:10:04.535224 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.535205 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="80391d09-653e-4056-aa8d-05768237b1c8" containerName="revoke-keys" Oct 30 13:10:04.540244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.540220 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:10:04.554000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.553879 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7e2137bc\"" Oct 30 13:10:04.554142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.554007 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7e2137bc-dockercfg-6nr5j\"" Oct 30 13:10:04.557774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.557673 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4"] Oct 30 13:10:04.560715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.559730 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:04.798960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.766135 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d980ebfb-c1ec-491f-b6a7-4b14025d9027-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4\" (UID: \"d980ebfb-c1ec-491f-b6a7-4b14025d9027\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:10:04.798960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.766241 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrr5h\" (UniqueName: \"kubernetes.io/projected/d980ebfb-c1ec-491f-b6a7-4b14025d9027-kube-api-access-lrr5h\") pod \"payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4\" (UID: \"d980ebfb-c1ec-491f-b6a7-4b14025d9027\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:10:04.896751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.896681 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d980ebfb-c1ec-491f-b6a7-4b14025d9027-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4\" (UID: \"d980ebfb-c1ec-491f-b6a7-4b14025d9027\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:10:04.967359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.959486 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrr5h\" (UniqueName: \"kubernetes.io/projected/d980ebfb-c1ec-491f-b6a7-4b14025d9027-kube-api-access-lrr5h\") pod \"payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4\" (UID: \"d980ebfb-c1ec-491f-b6a7-4b14025d9027\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:10:04.967359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.959979 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d980ebfb-c1ec-491f-b6a7-4b14025d9027-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4\" (UID: \"d980ebfb-c1ec-491f-b6a7-4b14025d9027\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:10:04.996431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:04.996397 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrr5h\" (UniqueName: \"kubernetes.io/projected/d980ebfb-c1ec-491f-b6a7-4b14025d9027-kube-api-access-lrr5h\") pod \"payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4\" (UID: \"d980ebfb-c1ec-491f-b6a7-4b14025d9027\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:10:05.000111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:05.000083 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:10:05.167179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:05.167091 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:10:05.176856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:05.175503 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:10:05.179972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:05.177350 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:10:05.180229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:05.180174 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:10:05.485438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:05.485336 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:05.570677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:05.570518 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:10:05.571525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:05.571501 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:10:05.572459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:05.572432 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:10:05.580371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:05.580347 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4"] Oct 30 13:10:05.582276 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:05.582249 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd980ebfb_c1ec_491f_b6a7_4b14025d9027.slice/crio-de02a9b98a46428388cd5deaf3e89bb9326ef5b3e597d7e99848cb1cc2fbb97f WatchSource:0}: Error finding container de02a9b98a46428388cd5deaf3e89bb9326ef5b3e597d7e99848cb1cc2fbb97f: Status 404 returned error can't find the container with id de02a9b98a46428388cd5deaf3e89bb9326ef5b3e597d7e99848cb1cc2fbb97f Oct 30 13:10:06.009720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:06.009685 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" event={"ID":"d980ebfb-c1ec-491f-b6a7-4b14025d9027","Type":"ContainerStarted","Data":"ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954"} Oct 30 13:10:06.010554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:06.009737 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" event={"ID":"d980ebfb-c1ec-491f-b6a7-4b14025d9027","Type":"ContainerStarted","Data":"de02a9b98a46428388cd5deaf3e89bb9326ef5b3e597d7e99848cb1cc2fbb97f"} Oct 30 13:10:06.010554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:06.010098 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:10:06.052216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:06.052158 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" podStartSLOduration=2.0521449 podStartE2EDuration="2.0521449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:06.04602565 +0000 UTC m=+1112.861220073" watchObservedRunningTime="2026-10-30 13:10:06.0521449 +0000 UTC m=+1112.867339323" Oct 30 13:10:07.073836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.073798 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h"] Oct 30 13:10:07.079469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.079437 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.098438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.098410 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h"] Oct 30 13:10:07.100484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.100461 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:07.104204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.104181 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:10:07.203419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.203387 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b8d54f-cqd6h\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.203681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.203658 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b8d54f-cqd6h\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.203756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.203715 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xqt\" (UniqueName: \"kubernetes.io/projected/b2ffede2-0aec-4064-a02b-fba1d717902c-kube-api-access-t2xqt\") pod \"maas-api-revoke-sub-c6b8d54f-cqd6h\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.298530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.295439 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:07.309689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.308373 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b8d54f-cqd6h\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.309689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.308446 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xqt\" (UniqueName: \"kubernetes.io/projected/b2ffede2-0aec-4064-a02b-fba1d717902c-kube-api-access-t2xqt\") pod \"maas-api-revoke-sub-c6b8d54f-cqd6h\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.309689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.308572 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b8d54f-cqd6h\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.311219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.311174 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b8d54f-cqd6h\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.313958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.313819 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b8d54f-cqd6h\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.338405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.336897 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xqt\" (UniqueName: \"kubernetes.io/projected/b2ffede2-0aec-4064-a02b-fba1d717902c-kube-api-access-t2xqt\") pod \"maas-api-revoke-sub-c6b8d54f-cqd6h\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.397832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.397742 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:07.728541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:07.728514 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h"] Oct 30 13:10:07.730953 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:07.730920 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ffede2_0aec_4064_a02b_fba1d717902c.slice/crio-b4e69230852bb2496e34316e59f097d44257be55fbadec5b9a44187473de7d45 WatchSource:0}: Error finding container b4e69230852bb2496e34316e59f097d44257be55fbadec5b9a44187473de7d45: Status 404 returned error can't find the container with id b4e69230852bb2496e34316e59f097d44257be55fbadec5b9a44187473de7d45 Oct 30 13:10:08.028986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:08.028959 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" event={"ID":"b2ffede2-0aec-4064-a02b-fba1d717902c","Type":"ContainerStarted","Data":"b4e69230852bb2496e34316e59f097d44257be55fbadec5b9a44187473de7d45"} Oct 30 13:10:08.073205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:08.073011 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" podStartSLOduration=1.072997209 podStartE2EDuration="1.072997209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:08.070917769 +0000 UTC m=+1114.886112260" watchObservedRunningTime="2026-10-30 13:10:08.072997209 +0000 UTC m=+1114.888191664" Oct 30 13:10:08.729136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:08.729101 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:09.039946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.039858 5049 generic.go:356] "Generic (PLEG): container finished" podID="b2ffede2-0aec-4064-a02b-fba1d717902c" containerID="25a6255b08128dbde9df2777099fe3b9357305c681e186c635d5bf377cba8241" exitCode=0 Oct 30 13:10:09.040084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.039939 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" event={"ID":"b2ffede2-0aec-4064-a02b-fba1d717902c","Type":"ContainerDied","Data":"25a6255b08128dbde9df2777099fe3b9357305c681e186c635d5bf377cba8241"} Oct 30 13:10:09.299399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.299362 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7"] Oct 30 13:10:09.304416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.304395 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.339196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.339160 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7"] Oct 30 13:10:09.447016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.446986 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28f44fd7-28mq7\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.447174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.447142 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28f44fd7-28mq7\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.447219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.447189 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp65t\" (UniqueName: \"kubernetes.io/projected/d33e1339-fd95-4d60-82e8-d534ef00b56e-kube-api-access-wp65t\") pod \"maas-api-revoke-sub-28f44fd7-28mq7\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.511492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.511422 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:09.550954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.549578 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28f44fd7-28mq7\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.550954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.549794 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28f44fd7-28mq7\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.550954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.549841 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp65t\" (UniqueName: \"kubernetes.io/projected/d33e1339-fd95-4d60-82e8-d534ef00b56e-kube-api-access-wp65t\") pod \"maas-api-revoke-sub-28f44fd7-28mq7\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.551255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.551157 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28f44fd7-28mq7\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.573933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.573900 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28f44fd7-28mq7\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.576579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.576553 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp65t\" (UniqueName: \"kubernetes.io/projected/d33e1339-fd95-4d60-82e8-d534ef00b56e-kube-api-access-wp65t\") pod \"maas-api-revoke-sub-28f44fd7-28mq7\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.618395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.618354 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:09.953938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:09.953910 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7"] Oct 30 13:10:09.954952 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:09.954677 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33e1339_fd95_4d60_82e8_d534ef00b56e.slice/crio-dc5697429c008450b30d13836c5e5a2544bef544187657815d6ee698a94cea6e WatchSource:0}: Error finding container dc5697429c008450b30d13836c5e5a2544bef544187657815d6ee698a94cea6e: Status 404 returned error can't find the container with id dc5697429c008450b30d13836c5e5a2544bef544187657815d6ee698a94cea6e Oct 30 13:10:10.051192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.051151 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" event={"ID":"d33e1339-fd95-4d60-82e8-d534ef00b56e","Type":"ContainerStarted","Data":"dc5697429c008450b30d13836c5e5a2544bef544187657815d6ee698a94cea6e"} Oct 30 13:10:10.214912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.214885 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:10.361547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.361519 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-xks-ca\") pod \"b2ffede2-0aec-4064-a02b-fba1d717902c\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " Oct 30 13:10:10.361704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.361627 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ffede2-0aec-4064-a02b-fba1d717902c-kube-api-access-t2xqt\" (UniqueName: \"kubernetes.io/projected/b2ffede2-0aec-4064-a02b-fba1d717902c-kube-api-access-t2xqt\") pod \"b2ffede2-0aec-4064-a02b-fba1d717902c\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " Oct 30 13:10:10.361704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.361655 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-openshift-ca\") pod \"b2ffede2-0aec-4064-a02b-fba1d717902c\" (UID: \"b2ffede2-0aec-4064-a02b-fba1d717902c\") " Oct 30 13:10:10.362122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.362090 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-openshift-ca" pod "b2ffede2-0aec-4064-a02b-fba1d717902c" (UID: "b2ffede2-0aec-4064-a02b-fba1d717902c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:10.362348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.362313 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:10.363662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.363637 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-xks-ca" pod "b2ffede2-0aec-4064-a02b-fba1d717902c" (UID: "b2ffede2-0aec-4064-a02b-fba1d717902c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:10.363839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.363764 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ffede2-0aec-4064-a02b-fba1d717902c-kube-api-access-t2xqt" pod "b2ffede2-0aec-4064-a02b-fba1d717902c" (UID: "b2ffede2-0aec-4064-a02b-fba1d717902c"). InnerVolumeSpecName "kube-api-access-t2xqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:10.464241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.464143 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2xqt\" (UniqueName: \"kubernetes.io/projected/b2ffede2-0aec-4064-a02b-fba1d717902c-kube-api-access-t2xqt\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:10.464241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:10.464170 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ffede2-0aec-4064-a02b-fba1d717902c-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.063585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:11.063556 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" Oct 30 13:10:11.063999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:11.063578 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h" event={"ID":"b2ffede2-0aec-4064-a02b-fba1d717902c","Type":"ContainerDied","Data":"b4e69230852bb2496e34316e59f097d44257be55fbadec5b9a44187473de7d45"} Oct 30 13:10:11.063999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:11.063637 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4e69230852bb2496e34316e59f097d44257be55fbadec5b9a44187473de7d45" Oct 30 13:10:11.065298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:11.065262 5049 generic.go:356] "Generic (PLEG): container finished" podID="d33e1339-fd95-4d60-82e8-d534ef00b56e" containerID="434dc04da05ac3c7bbcc56be5ea106621615ebfc1bdcbc4f3e3085f429af0243" exitCode=0 Oct 30 13:10:11.065454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:11.065347 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" event={"ID":"d33e1339-fd95-4d60-82e8-d534ef00b56e","Type":"ContainerDied","Data":"434dc04da05ac3c7bbcc56be5ea106621615ebfc1bdcbc4f3e3085f429af0243"} Oct 30 13:10:11.461676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:11.461640 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:12.270040 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.270011 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:12.391974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.391879 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-xks-ca\") pod \"d33e1339-fd95-4d60-82e8-d534ef00b56e\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " Oct 30 13:10:12.392129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.392028 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33e1339-fd95-4d60-82e8-d534ef00b56e-kube-api-access-wp65t\" (UniqueName: \"kubernetes.io/projected/d33e1339-fd95-4d60-82e8-d534ef00b56e-kube-api-access-wp65t\") pod \"d33e1339-fd95-4d60-82e8-d534ef00b56e\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " Oct 30 13:10:12.392129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.392086 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-openshift-ca\") pod \"d33e1339-fd95-4d60-82e8-d534ef00b56e\" (UID: \"d33e1339-fd95-4d60-82e8-d534ef00b56e\") " Oct 30 13:10:12.392477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.392449 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-openshift-ca" pod "d33e1339-fd95-4d60-82e8-d534ef00b56e" (UID: "d33e1339-fd95-4d60-82e8-d534ef00b56e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:12.392735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.392715 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:12.394262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.394241 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33e1339-fd95-4d60-82e8-d534ef00b56e-kube-api-access-wp65t" pod "d33e1339-fd95-4d60-82e8-d534ef00b56e" (UID: "d33e1339-fd95-4d60-82e8-d534ef00b56e"). InnerVolumeSpecName "kube-api-access-wp65t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:12.394538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.394517 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-xks-ca" pod "d33e1339-fd95-4d60-82e8-d534ef00b56e" (UID: "d33e1339-fd95-4d60-82e8-d534ef00b56e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:12.494087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.494058 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp65t\" (UniqueName: \"kubernetes.io/projected/d33e1339-fd95-4d60-82e8-d534ef00b56e-kube-api-access-wp65t\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:12.494087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.494084 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33e1339-fd95-4d60-82e8-d534ef00b56e-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:12.803707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:12.803673 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:13.083889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:13.083805 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" event={"ID":"d33e1339-fd95-4d60-82e8-d534ef00b56e","Type":"ContainerDied","Data":"dc5697429c008450b30d13836c5e5a2544bef544187657815d6ee698a94cea6e"} Oct 30 13:10:13.083889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:13.083864 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc5697429c008450b30d13836c5e5a2544bef544187657815d6ee698a94cea6e" Oct 30 13:10:13.083889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:13.083817 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7" Oct 30 13:10:13.640186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:13.640117 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:14.645424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.645341 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c"] Oct 30 13:10:14.646410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.646389 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d33e1339-fd95-4d60-82e8-d534ef00b56e" containerName="revoke-keys" Oct 30 13:10:14.646489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.646413 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33e1339-fd95-4d60-82e8-d534ef00b56e" containerName="revoke-keys" Oct 30 13:10:14.646489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.646477 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b2ffede2-0aec-4064-a02b-fba1d717902c" containerName="revoke-keys" Oct 30 13:10:14.646489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.646486 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ffede2-0aec-4064-a02b-fba1d717902c" containerName="revoke-keys" Oct 30 13:10:14.646691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.646679 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33e1339-fd95-4d60-82e8-d534ef00b56e" containerName="revoke-keys" Oct 30 13:10:14.646723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.646703 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ffede2-0aec-4064-a02b-fba1d717902c" containerName="revoke-keys" Oct 30 13:10:14.651563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.651535 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:14.668930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.668908 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:14.677003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.676973 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c"] Oct 30 13:10:14.680106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.680086 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:10:14.844281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.844242 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e785762-kxk2c\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:14.844490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.844334 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e785762-kxk2c\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:14.844645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.844615 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2c7l\" (UniqueName: \"kubernetes.io/projected/131e791f-b22d-455d-bfe2-e93e3fc3efb7-kube-api-access-c2c7l\") pod \"maas-api-revoke-sub-3e785762-kxk2c\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:14.945658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.945560 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2c7l\" (UniqueName: \"kubernetes.io/projected/131e791f-b22d-455d-bfe2-e93e3fc3efb7-kube-api-access-c2c7l\") pod \"maas-api-revoke-sub-3e785762-kxk2c\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:14.945821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.945669 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e785762-kxk2c\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:14.945821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.945728 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e785762-kxk2c\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:14.946507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.946479 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e785762-kxk2c\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:14.948211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.948187 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e785762-kxk2c\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:14.968577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:14.968553 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2c7l\" (UniqueName: \"kubernetes.io/projected/131e791f-b22d-455d-bfe2-e93e3fc3efb7-kube-api-access-c2c7l\") pod \"maas-api-revoke-sub-3e785762-kxk2c\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:15.265517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:15.265433 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:15.649889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:15.646030 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:15.649889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:15.646736 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c"] Oct 30 13:10:15.830726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:15.830685 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:16.012937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:16.012910 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:10:16.118184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:16.118146 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" event={"ID":"131e791f-b22d-455d-bfe2-e93e3fc3efb7","Type":"ContainerStarted","Data":"007e92f0cd4743c5720a264d00b1b1c7f22698a3ed05b1ef066246b6d368ab92"} Oct 30 13:10:16.118373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:16.118203 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" event={"ID":"131e791f-b22d-455d-bfe2-e93e3fc3efb7","Type":"ContainerStarted","Data":"a103b84b813dfb9d38e2f05ab6dc1335c4ac32483464ee8a2d670237f4e70dd4"} Oct 30 13:10:16.180453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:16.180419 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw"] Oct 30 13:10:16.192823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:16.192795 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3037e700-69drw"] Oct 30 13:10:16.203078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:16.203031 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" podStartSLOduration=2.203018192 podStartE2EDuration="2.203018192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:16.19915704 +0000 UTC m=+1123.014351462" watchObservedRunningTime="2026-10-30 13:10:16.203018192 +0000 UTC m=+1123.018212615" Oct 30 13:10:17.020981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.020902 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:10:17.132282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.132244 5049 generic.go:356] "Generic (PLEG): container finished" podID="131e791f-b22d-455d-bfe2-e93e3fc3efb7" containerID="007e92f0cd4743c5720a264d00b1b1c7f22698a3ed05b1ef066246b6d368ab92" exitCode=0 Oct 30 13:10:17.132607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.132578 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" event={"ID":"131e791f-b22d-455d-bfe2-e93e3fc3efb7","Type":"ContainerDied","Data":"007e92f0cd4743c5720a264d00b1b1c7f22698a3ed05b1ef066246b6d368ab92"} Oct 30 13:10:17.341536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.341500 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a04399-e68c-4978-981b-f2987aec0e16" path="/var/lib/kubelet/pods/c2a04399-e68c-4978-981b-f2987aec0e16/volumes" Oct 30 13:10:17.547268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.547229 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd"] Oct 30 13:10:17.552436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.552414 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.573486 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.573455 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b31c2926-serving-cert\"" Oct 30 13:10:17.579000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.578973 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd"] Oct 30 13:10:17.604425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.598704 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-metrics-certs\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.604425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.598775 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-maas-api-tls\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.604425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.598868 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d694fe15-a212-46e5-941f-14b9f63da7a3-openshift-service-ca\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.604425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.599000 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/d694fe15-a212-46e5-941f-14b9f63da7a3-kube-api-access-gxwxt\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.700226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.700194 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-metrics-certs\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.700415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.700245 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-maas-api-tls\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.700415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.700305 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d694fe15-a212-46e5-941f-14b9f63da7a3-openshift-service-ca\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.700488 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:10:17.700410 5049 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-serving-cert: secret "maas-api-e2e-ait-b31c2926-serving-cert" not found Oct 30 13:10:17.700488 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:10:17.700481 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-maas-api-tls podName:d694fe15-a212-46e5-941f-14b9f63da7a3 nodeName:}" failed. No retries permitted until 2026-10-30 13:10:18.200465513 +0000 UTC m=+1125.015659919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-maas-api-tls") pod "maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" (UID: "d694fe15-a212-46e5-941f-14b9f63da7a3") : secret "maas-api-e2e-ait-b31c2926-serving-cert" not found Oct 30 13:10:17.700597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.700415 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/d694fe15-a212-46e5-941f-14b9f63da7a3-kube-api-access-gxwxt\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.701034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.701013 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d694fe15-a212-46e5-941f-14b9f63da7a3-openshift-service-ca\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.702880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.702856 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-metrics-certs\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.708525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.708500 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:17.731773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.731742 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/d694fe15-a212-46e5-941f-14b9f63da7a3-kube-api-access-gxwxt\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:17.872895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:17.872797 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:18.050247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.050205 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:18.213100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.212942 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-maas-api-tls\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:18.235814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.235735 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-maas-api-tls\") pod \"maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:18.331556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.331519 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:18.415643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.415539 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-xks-ca\") pod \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " Oct 30 13:10:18.415643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.415599 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131e791f-b22d-455d-bfe2-e93e3fc3efb7-kube-api-access-c2c7l\" (UniqueName: \"kubernetes.io/projected/131e791f-b22d-455d-bfe2-e93e3fc3efb7-kube-api-access-c2c7l\") pod \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " Oct 30 13:10:18.416255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.415910 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-openshift-ca\") pod \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\" (UID: \"131e791f-b22d-455d-bfe2-e93e3fc3efb7\") " Oct 30 13:10:18.416445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.416419 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-openshift-ca" pod "131e791f-b22d-455d-bfe2-e93e3fc3efb7" (UID: "131e791f-b22d-455d-bfe2-e93e3fc3efb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:18.417491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.416847 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:18.418497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.418476 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-xks-ca" pod "131e791f-b22d-455d-bfe2-e93e3fc3efb7" (UID: "131e791f-b22d-455d-bfe2-e93e3fc3efb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:18.419035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.419017 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131e791f-b22d-455d-bfe2-e93e3fc3efb7-kube-api-access-c2c7l" pod "131e791f-b22d-455d-bfe2-e93e3fc3efb7" (UID: "131e791f-b22d-455d-bfe2-e93e3fc3efb7"). InnerVolumeSpecName "kube-api-access-c2c7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:18.467952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.467914 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:18.518295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.518263 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/131e791f-b22d-455d-bfe2-e93e3fc3efb7-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:18.518295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.518293 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2c7l\" (UniqueName: \"kubernetes.io/projected/131e791f-b22d-455d-bfe2-e93e3fc3efb7-kube-api-access-c2c7l\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:18.855732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.855701 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd"] Oct 30 13:10:18.856090 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:18.856018 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd694fe15_a212_46e5_941f_14b9f63da7a3.slice/crio-1e4c5dc2a5af4280d0a6df646322f3464b8369bde5df68e21de7ade7be652092 WatchSource:0}: Error finding container 1e4c5dc2a5af4280d0a6df646322f3464b8369bde5df68e21de7ade7be652092: Status 404 returned error can't find the container with id 1e4c5dc2a5af4280d0a6df646322f3464b8369bde5df68e21de7ade7be652092 Oct 30 13:10:18.858164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:18.858144 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:10:19.164390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:19.162503 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" event={"ID":"131e791f-b22d-455d-bfe2-e93e3fc3efb7","Type":"ContainerDied","Data":"a103b84b813dfb9d38e2f05ab6dc1335c4ac32483464ee8a2d670237f4e70dd4"} Oct 30 13:10:19.164390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:19.162568 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a103b84b813dfb9d38e2f05ab6dc1335c4ac32483464ee8a2d670237f4e70dd4" Oct 30 13:10:19.164390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:19.162644 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c" Oct 30 13:10:19.166980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:19.166934 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" event={"ID":"d694fe15-a212-46e5-941f-14b9f63da7a3","Type":"ContainerStarted","Data":"1e4c5dc2a5af4280d0a6df646322f3464b8369bde5df68e21de7ade7be652092"} Oct 30 13:10:19.483016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:19.482917 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:19.652873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:19.652831 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:19.816251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:19.816215 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:19.989628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:19.988860 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:20.472864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:20.472817 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" event={"ID":"d694fe15-a212-46e5-941f-14b9f63da7a3","Type":"ContainerStarted","Data":"040a782035e0c59411a77613dbd6b4afc3e0c952404086c3c239d4a3a2aef0ea"} Oct 30 13:10:20.474902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:20.474877 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:21.625344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:21.624669 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:21.679503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:21.679456 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:22.140345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.138419 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:22.179571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.179523 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:22.206191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.206148 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:22.400597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.400446 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:10:22.626477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.626433 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:22.627403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.627369 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:10:22.627880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.627855 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:10:22.628720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.628700 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:10:22.683807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.683700 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:22.685345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.684307 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:10:22.721015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.720951 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" podStartSLOduration=5.367718022 podStartE2EDuration="5.720929714s" totalImagesPullingTime="353.211692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:20.542420668 +0000 UTC m=+1127.357615091" watchObservedRunningTime="2026-10-30 13:10:22.720929714 +0000 UTC m=+1129.536124134" Oct 30 13:10:22.892476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:22.890672 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:10:23.215907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.207418 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.229549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.229502 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:10:23.310208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.277521 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.310887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.277556 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:10:23.332674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.331051 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:10:23.387345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.386422 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:10:23.484390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.409155 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.619497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.619444 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.650350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.628799 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.662751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.651135 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:10:23.662751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.657121 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:10:23.779512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.713316 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:10:23.838353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.800793 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:10:23.853885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.851928 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh"] Oct 30 13:10:23.887581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.887538 5049 cpu_manager.go:414] "containerMap: removing container" podUID="131e791f-b22d-455d-bfe2-e93e3fc3efb7" containerName="revoke-keys" Oct 30 13:10:23.946334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.946280 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131e791f-b22d-455d-bfe2-e93e3fc3efb7" containerName="revoke-keys" Oct 30 13:10:23.947625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.947599 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="131e791f-b22d-455d-bfe2-e93e3fc3efb7" containerName="revoke-keys" Oct 30 13:10:23.965403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:23.898979 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.057408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.051942 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.207078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.197551 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh"] Oct 30 13:10:24.207078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.197655 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:10:24.207078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.197701 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:10:24.207078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.197738 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:10:24.207078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.198436 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.222346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.207957 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.327436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.315241 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.327436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.316086 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:10:24.344346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.340898 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.347862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.344565 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.348109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.348086 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.355389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.355357 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.355668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.355648 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.458345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.450135 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.533204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.526370 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.533204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.526421 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.533204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.526472 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.533204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.526598 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.533204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.526630 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.533204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.526719 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.533204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.526776 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.533204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.526808 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk24m\" (UniqueName: \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-kube-api-access-dk24m\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.533204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.526838 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.534360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.534310 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.534832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.534807 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.549445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.535207 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.559357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.546888 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.635412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.635269 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.651736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.651693 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.674119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.673170 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:10:24.674119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.673826 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.674119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.674023 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.674119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.674060 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk24m\" (UniqueName: \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-kube-api-access-dk24m\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.674119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.674107 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.712346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.680164 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:24.728874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.728822 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:10:25.225929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.807402 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz"] Oct 30 13:10:25.234460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.227681 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:25.341358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.975946 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:25.455391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.235541 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:25.455391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.236533 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk24m\" (UniqueName: \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-kube-api-access-dk24m\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:25.455391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.237006 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.460677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.460634 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.506990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.499956 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.506990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.506039 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.506990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.372658 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 13:10:25.531349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.521947 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:25.719426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.644038 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:10:25.719426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:25.671514 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.719426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:24.975482 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.045802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.045759 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.103340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.097066 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:10:26.349501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.349460 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:10:26.419067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.401374 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:26.512350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.502760 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:10:26.519368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.513176 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.553627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.553585 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.696763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.696667 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:26.799209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.799176 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:26.800835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.800783 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:26.801086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.801066 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2rtk\" (UniqueName: \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-kube-api-access-q2rtk\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:26.801394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.801373 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:26.890310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.795818 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:26.896494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.747388 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.896494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.880238 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:10:26.919616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.887524 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:26.921360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.921310 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:26.922172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.922151 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:26.923181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.923142 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:26.944208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.939412 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.945056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.944669 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8"] Oct 30 13:10:26.959008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.958920 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz"] Oct 30 13:10:26.959008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.958996 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8"] Oct 30 13:10:26.959221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.959082 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:10:26.959221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.959142 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:10:26.959221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.959198 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:10:26.959413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.959234 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:10:26.959413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.959297 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:10:26.959413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.959388 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:10:26.959582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.959473 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:10:26.962109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:26.962086 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:26.998442 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:10:26.998411 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:10:27.029139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029099 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.029360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029160 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.029360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029263 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2rtk\" (UniqueName: \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-kube-api-access-q2rtk\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.029490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029394 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.029546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029498 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.029597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029560 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.029653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029638 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.029714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029673 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.029771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029734 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4mcb\" (UniqueName: \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-kube-api-access-l4mcb\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.029827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029817 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.029904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.029886 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.030075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.030059 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.030156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.030120 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.030216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.030180 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.030273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.030241 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.042600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.042563 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.042770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.042676 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.042770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.042714 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.043680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.043643 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.044462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.044251 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.047723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.047696 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.052145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.049422 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.053720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.053693 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.064572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.064549 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.065527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.065504 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.069610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.069580 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2rtk\" (UniqueName: \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-kube-api-access-q2rtk\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.077031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.077012 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.080813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.080783 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:27.144762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.144676 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.144982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.144965 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.145267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.145251 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.145400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.145386 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.145522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.145507 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4mcb\" (UniqueName: \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-kube-api-access-l4mcb\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.145649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.145635 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.145740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.145711 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.145935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.145430 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.146043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.145746 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.146223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.146206 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.146346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.146314 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.146430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.146409 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.146563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.146537 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.146971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.146946 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.147250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.147225 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.148904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.148886 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.178227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.178187 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.178825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.178800 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh"] Oct 30 13:10:27.179109 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:27.178879 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951a9e8c_d9b8_47a8_a02a_56c94cea5bba.slice/crio-46a73b8a28244106b89f12cfde248610b7cda13771840f1cb1aa2770b40e9778 WatchSource:0}: Error finding container 46a73b8a28244106b89f12cfde248610b7cda13771840f1cb1aa2770b40e9778: Status 404 returned error can't find the container with id 46a73b8a28244106b89f12cfde248610b7cda13771840f1cb1aa2770b40e9778 Oct 30 13:10:27.180750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.180719 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4mcb\" (UniqueName: \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-kube-api-access-l4mcb\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.182952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.182910 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.183145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.183119 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.183254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.183190 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.272622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.272590 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:27.302809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.302600 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:27.310271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.310165 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:27.509605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.509567 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:27.548753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.548645 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:10:27.676892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.676864 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz"] Oct 30 13:10:27.680126 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:27.680101 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7f05d7_e738_44ca_93ed_f5c02d64084a.slice/crio-ab4987a82a61996ddf804a99101957123dcd3cc5027baba01cc6669224993519 WatchSource:0}: Error finding container ab4987a82a61996ddf804a99101957123dcd3cc5027baba01cc6669224993519: Status 404 returned error can't find the container with id ab4987a82a61996ddf804a99101957123dcd3cc5027baba01cc6669224993519 Oct 30 13:10:27.683801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.683759 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.683916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.683841 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.683916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.683891 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.736963 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:27.736927 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2f7fb4_e4bd_49c4_a070_bed24a63733e.slice/crio-9f8419d3fa9fb109420ca5f4bf312037f5fe69426e9e560992da54bb0c9220d8 WatchSource:0}: Error finding container 9f8419d3fa9fb109420ca5f4bf312037f5fe69426e9e560992da54bb0c9220d8: Status 404 returned error can't find the container with id 9f8419d3fa9fb109420ca5f4bf312037f5fe69426e9e560992da54bb0c9220d8 Oct 30 13:10:27.743408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.743086 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8"] Oct 30 13:10:27.746719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.746681 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.746828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.746762 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.746828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.746809 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.882834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.882796 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" event={"ID":"951a9e8c-d9b8-47a8-a02a-56c94cea5bba","Type":"ContainerStarted","Data":"4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630"} Oct 30 13:10:27.883221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.882865 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" event={"ID":"951a9e8c-d9b8-47a8-a02a-56c94cea5bba","Type":"ContainerStarted","Data":"46a73b8a28244106b89f12cfde248610b7cda13771840f1cb1aa2770b40e9778"} Oct 30 13:10:27.886112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.886084 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" event={"ID":"fc7f05d7-e738-44ca-93ed-f5c02d64084a","Type":"ContainerStarted","Data":"e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571"} Oct 30 13:10:27.886230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.886130 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" event={"ID":"fc7f05d7-e738-44ca-93ed-f5c02d64084a","Type":"ContainerStarted","Data":"ab4987a82a61996ddf804a99101957123dcd3cc5027baba01cc6669224993519"} Oct 30 13:10:27.888198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.888171 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" event={"ID":"9f2f7fb4-e4bd-49c4-a070-bed24a63733e","Type":"ContainerStarted","Data":"ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080"} Oct 30 13:10:27.888270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.888219 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" event={"ID":"9f2f7fb4-e4bd-49c4-a070-bed24a63733e","Type":"ContainerStarted","Data":"9f8419d3fa9fb109420ca5f4bf312037f5fe69426e9e560992da54bb0c9220d8"} Oct 30 13:10:27.971132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:27.971011 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" podStartSLOduration=4.970992678 podStartE2EDuration="4.970992678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:27.960220212 +0000 UTC m=+1134.775414660" watchObservedRunningTime="2026-10-30 13:10:27.970992678 +0000 UTC m=+1134.786187103" Oct 30 13:10:28.031393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.031332 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" podStartSLOduration=5.031301303 podStartE2EDuration="5.031301303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:28.030943967 +0000 UTC m=+1134.846138391" watchObservedRunningTime="2026-10-30 13:10:28.031301303 +0000 UTC m=+1134.846495728" Oct 30 13:10:28.085189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.085138 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" podStartSLOduration=5.085122328 podStartE2EDuration="5.085122328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:28.081137983 +0000 UTC m=+1134.896332406" watchObservedRunningTime="2026-10-30 13:10:28.085122328 +0000 UTC m=+1134.900316752" Oct 30 13:10:28.292732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.292648 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:28.292963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.292828 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:10:28.293105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.293082 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 13:10:28.303448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.303419 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:28.304817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.304792 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:10:28.304918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.304843 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 13:10:28.497067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.497020 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:28.616958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.616918 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:28.641498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.641459 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 13:10:28.641792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:28.641760 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 13:10:29.284958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:29.284905 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:10:29.285549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:29.284979 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 13:10:29.303590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:29.303546 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:10:29.303856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:29.303829 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 13:10:29.607347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:29.604653 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 13:10:29.607347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:29.604714 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 13:10:29.923528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:29.906107 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz"] Oct 30 13:10:30.089350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.063838 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.195407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.179734 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:10:30.195407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.179808 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:30.267385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.228284 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20713d97-v5dxz\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.267385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.228420 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7km5\" (UniqueName: \"kubernetes.io/projected/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-kube-api-access-w7km5\") pod \"maas-api-revoke-sub-20713d97-v5dxz\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.267385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.228622 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20713d97-v5dxz\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.291345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.287735 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz"] Oct 30 13:10:30.291345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.287928 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:10:30.291345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.287971 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 13:10:30.331190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.331153 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:10:30.373977 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.373931 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 13:10:30.374208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.332350 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7km5\" (UniqueName: \"kubernetes.io/projected/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-kube-api-access-w7km5\") pod \"maas-api-revoke-sub-20713d97-v5dxz\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.374736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.374708 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20713d97-v5dxz\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.375024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.374960 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20713d97-v5dxz\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.410619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.386588 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20713d97-v5dxz\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.411391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.411362 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:30.435017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.434983 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20713d97-v5dxz\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.460412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.446989 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7km5\" (UniqueName: \"kubernetes.io/projected/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-kube-api-access-w7km5\") pod \"maas-api-revoke-sub-20713d97-v5dxz\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.534845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.534809 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:30.668342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.668157 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 13:10:30.668342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.668228 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 13:10:30.760417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:30.753917 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:31.330535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:31.328835 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:10:31.330535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:31.328891 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 13:10:31.330535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:31.329263 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:10:31.407283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:31.406458 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:10:31.407283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:31.406526 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 13:10:31.606344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:31.606244 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 13:10:31.606919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:31.606554 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 13:10:31.781357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:31.781291 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz"] Oct 30 13:10:31.890046 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:31.889997 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb568f4f7_53d1_49ee_9ec8_0fb9c51e3f2d.slice/crio-b4a6751c73f2a79190ede15e2c0adb6795339a41947f5deee60fe46b87ad0a91 WatchSource:0}: Error finding container b4a6751c73f2a79190ede15e2c0adb6795339a41947f5deee60fe46b87ad0a91: Status 404 returned error can't find the container with id b4a6751c73f2a79190ede15e2c0adb6795339a41947f5deee60fe46b87ad0a91 Oct 30 13:10:31.986438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:31.971108 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:32.110390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.110347 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:32.302557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.302067 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:10:32.302557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.302132 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 13:10:32.310546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.310438 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:10:32.310546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.310491 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 13:10:32.379461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.379414 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:32.387344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.381536 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:10:32.471352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.462442 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:32.697061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.697010 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:32.771001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.770961 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 13:10:32.771271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.771246 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 13:10:32.815460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.808638 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:32.986193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:32.986094 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" event={"ID":"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d","Type":"ContainerStarted","Data":"b4a6751c73f2a79190ede15e2c0adb6795339a41947f5deee60fe46b87ad0a91"} Oct 30 13:10:33.227460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.219305 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:10:33.238124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.237605 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:33.293699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.293657 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:10:33.293877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.293722 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 13:10:33.456433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.455966 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:10:33.456433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.456042 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 13:10:33.617538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.617499 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 13:10:33.617862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.617819 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 13:10:33.694569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.694524 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:33.728565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.697421 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:33.757340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.757189 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:33.780381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.760497 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:33.794877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:33.794849 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:10:34.030565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.030469 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:34.071279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.071233 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:34.224344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.223415 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:34.251342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.250371 5049 generic.go:356] "Generic (PLEG): container finished" podID="b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d" containerID="958fca6ec2af7c1dbddbc2d1d5c93de4f641e61bdfaa7891f2b348fb00f0fbf0" exitCode=0 Oct 30 13:10:34.251342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.250744 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" event={"ID":"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d","Type":"ContainerDied","Data":"958fca6ec2af7c1dbddbc2d1d5c93de4f641e61bdfaa7891f2b348fb00f0fbf0"} Oct 30 13:10:34.279341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.279104 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:10:34.279341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.279178 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 13:10:34.345346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.344558 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:10:34.345346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.344613 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 13:10:34.444344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.443843 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch"] Oct 30 13:10:34.497342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.496303 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:10:34.513118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.502059 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f1e5fd-ht4ch"] Oct 30 13:10:34.574386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.571463 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:34.613922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.613831 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 13:10:34.614162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:34.614135 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 13:10:35.274017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.273978 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:10:35.274188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.274062 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 13:10:35.309872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.309833 5049 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 13:10:35.310089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.309906 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 13:10:35.355922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.347701 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec47a3e-9af4-4396-bc97-85c8ae728b92" path="/var/lib/kubelet/pods/aec47a3e-9af4-4396-bc97-85c8ae728b92/volumes" Oct 30 13:10:35.520815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.520752 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:35.606375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.604505 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 13:10:35.606375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.604566 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 13:10:35.651660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.651623 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-kube-api-access-w7km5\" (UniqueName: \"kubernetes.io/projected/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-kube-api-access-w7km5\") pod \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " Oct 30 13:10:35.651924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.651905 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-xks-ca\") pod \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " Oct 30 13:10:35.652000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.651968 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-openshift-ca\") pod \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\" (UID: \"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d\") " Oct 30 13:10:35.652861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.652824 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-openshift-ca" pod "b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d" (UID: "b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:35.654105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.654078 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-kube-api-access-w7km5" pod "b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d" (UID: "b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d"). InnerVolumeSpecName "kube-api-access-w7km5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:35.654975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.654949 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-xks-ca" pod "b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d" (UID: "b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:35.756373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.755343 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:35.756373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.755427 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:35.756373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:35.755446 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7km5\" (UniqueName: \"kubernetes.io/projected/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d-kube-api-access-w7km5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:36.308342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.308138 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" event={"ID":"b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d","Type":"ContainerDied","Data":"b4a6751c73f2a79190ede15e2c0adb6795339a41947f5deee60fe46b87ad0a91"} Oct 30 13:10:36.308342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.308251 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4a6751c73f2a79190ede15e2c0adb6795339a41947f5deee60fe46b87ad0a91" Oct 30 13:10:36.308608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.308375 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz" Oct 30 13:10:36.342418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.334296 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:36.522345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.521912 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:36.611135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.611050 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:36.612765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.612739 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:36.613574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.613547 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:10:36.700492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.700455 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:36.701218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.701196 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:36.708351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.703145 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:10:36.834368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.831705 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:36.834368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.833088 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:36.834368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:36.834039 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:10:37.042550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:37.042475 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx"] Oct 30 13:10:37.062900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:37.058273 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff3c51a-kndmx"] Oct 30 13:10:37.352351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:37.352280 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1231f0e6-4e20-4f34-b004-b7a284cf26d8" path="/var/lib/kubelet/pods/1231f0e6-4e20-4f34-b004-b7a284cf26d8/volumes" Oct 30 13:10:38.298961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:38.298924 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:40.256783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:40.256747 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:40.462886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:40.462565 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:40.668146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:40.668094 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:42.347087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:42.347041 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:43.033092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:43.033054 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8"] Oct 30 13:10:43.042894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:43.042863 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ae9417-p8zq8"] Oct 30 13:10:43.283862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:43.283783 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:43.342110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:43.342071 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6721e397-4f87-450a-b39f-502c4bb65e9b" path="/var/lib/kubelet/pods/6721e397-4f87-450a-b39f-502c4bb65e9b/volumes" Oct 30 13:10:43.743017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:43.742969 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:44.055206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.055157 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f"] Oct 30 13:10:44.056557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.056531 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d" containerName="revoke-keys" Oct 30 13:10:44.056709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.056695 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d" containerName="revoke-keys" Oct 30 13:10:44.057028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.057014 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d" containerName="revoke-keys" Oct 30 13:10:44.099471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.095958 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f"] Oct 30 13:10:44.099471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.096073 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.123350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.120270 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:44.135556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.135522 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:10:44.210649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.210611 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5dace6-dms2f\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.211012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.210990 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5dace6-dms2f\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.211166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.211151 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zggfp\" (UniqueName: \"kubernetes.io/projected/d930d022-36f9-4f16-9c29-f114847a9d3a-kube-api-access-zggfp\") pod \"maas-api-revoke-sub-4f5dace6-dms2f\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.316109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.316004 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5dace6-dms2f\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.316109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.316059 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zggfp\" (UniqueName: \"kubernetes.io/projected/d930d022-36f9-4f16-9c29-f114847a9d3a-kube-api-access-zggfp\") pod \"maas-api-revoke-sub-4f5dace6-dms2f\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.316562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.316512 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5dace6-dms2f\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.317660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.317631 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5dace6-dms2f\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.319907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.319876 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5dace6-dms2f\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.341962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.341930 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zggfp\" (UniqueName: \"kubernetes.io/projected/d930d022-36f9-4f16-9c29-f114847a9d3a-kube-api-access-zggfp\") pod \"maas-api-revoke-sub-4f5dace6-dms2f\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.526926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.526895 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:44.658248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.658155 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:44.927511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:44.927476 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f"] Oct 30 13:10:44.935748 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:44.935711 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd930d022_36f9_4f16_9c29_f114847a9d3a.slice/crio-588c29fed55a674a2e58fcba0009913db1c86b26c6d9aeb3b96b79ede6b10e7f WatchSource:0}: Error finding container 588c29fed55a674a2e58fcba0009913db1c86b26c6d9aeb3b96b79ede6b10e7f: Status 404 returned error can't find the container with id 588c29fed55a674a2e58fcba0009913db1c86b26c6d9aeb3b96b79ede6b10e7f Oct 30 13:10:45.436492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:45.436453 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" event={"ID":"d930d022-36f9-4f16-9c29-f114847a9d3a","Type":"ContainerStarted","Data":"9946a0cab3d8c12a0f43ef72cb64f780f7cc06d11d8040e0228fcf9bb87c94b8"} Oct 30 13:10:45.436688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:45.436509 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" event={"ID":"d930d022-36f9-4f16-9c29-f114847a9d3a","Type":"ContainerStarted","Data":"588c29fed55a674a2e58fcba0009913db1c86b26c6d9aeb3b96b79ede6b10e7f"} Oct 30 13:10:45.484278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:45.484223 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" podStartSLOduration=1.484209526 podStartE2EDuration="1.484209526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:45.475957102 +0000 UTC m=+1152.291151530" watchObservedRunningTime="2026-10-30 13:10:45.484209526 +0000 UTC m=+1152.299403947" Oct 30 13:10:46.449414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:46.449373 5049 generic.go:356] "Generic (PLEG): container finished" podID="d930d022-36f9-4f16-9c29-f114847a9d3a" containerID="9946a0cab3d8c12a0f43ef72cb64f780f7cc06d11d8040e0228fcf9bb87c94b8" exitCode=0 Oct 30 13:10:46.449803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:46.449610 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" event={"ID":"d930d022-36f9-4f16-9c29-f114847a9d3a","Type":"ContainerDied","Data":"9946a0cab3d8c12a0f43ef72cb64f780f7cc06d11d8040e0228fcf9bb87c94b8"} Oct 30 13:10:46.699790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:46.693718 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:47.037485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.037404 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc"] Oct 30 13:10:47.042569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.042544 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.047746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.047396 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9"] Oct 30 13:10:47.055650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.055623 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea114b-dd2s9"] Oct 30 13:10:47.058432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.058406 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.058576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.058559 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.058724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.058707 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqmxh\" (UniqueName: \"kubernetes.io/projected/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-kube-api-access-hqmxh\") pod \"maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.071996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.071964 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc"] Oct 30 13:10:47.159664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.159625 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.159833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.159739 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmxh\" (UniqueName: \"kubernetes.io/projected/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-kube-api-access-hqmxh\") pod \"maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.159833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.159806 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.160423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.160400 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.161963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.161944 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.184194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.184152 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmxh\" (UniqueName: \"kubernetes.io/projected/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-kube-api-access-hqmxh\") pod \"maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.342729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.342687 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31fc3bef-c3c7-4820-8118-c76a37e8df47" path="/var/lib/kubelet/pods/31fc3bef-c3c7-4820-8118-c76a37e8df47/volumes" Oct 30 13:10:47.370372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.362125 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:47.684825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.684793 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:47.685579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.685529 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:47.688980 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:47.688958 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e90e5c8_ee4d_4ae5_9f2e_4be771ef52d3.slice/crio-1a3c9fd1dee341abf47f8b269f8eb2e288cecb319c0c7c0445dd6d786d9259a1 WatchSource:0}: Error finding container 1a3c9fd1dee341abf47f8b269f8eb2e288cecb319c0c7c0445dd6d786d9259a1: Status 404 returned error can't find the container with id 1a3c9fd1dee341abf47f8b269f8eb2e288cecb319c0c7c0445dd6d786d9259a1 Oct 30 13:10:47.689521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.689498 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc"] Oct 30 13:10:47.771166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.771136 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d930d022-36f9-4f16-9c29-f114847a9d3a-kube-api-access-zggfp\" (UniqueName: \"kubernetes.io/projected/d930d022-36f9-4f16-9c29-f114847a9d3a-kube-api-access-zggfp\") pod \"d930d022-36f9-4f16-9c29-f114847a9d3a\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " Oct 30 13:10:47.771350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.771207 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-openshift-ca\") pod \"d930d022-36f9-4f16-9c29-f114847a9d3a\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " Oct 30 13:10:47.771405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.771375 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-xks-ca\") pod \"d930d022-36f9-4f16-9c29-f114847a9d3a\" (UID: \"d930d022-36f9-4f16-9c29-f114847a9d3a\") " Oct 30 13:10:47.771748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.771704 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-openshift-ca" pod "d930d022-36f9-4f16-9c29-f114847a9d3a" (UID: "d930d022-36f9-4f16-9c29-f114847a9d3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:47.771931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.771909 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.775059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.773621 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-xks-ca" pod "d930d022-36f9-4f16-9c29-f114847a9d3a" (UID: "d930d022-36f9-4f16-9c29-f114847a9d3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:47.775059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.773864 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d930d022-36f9-4f16-9c29-f114847a9d3a-kube-api-access-zggfp" pod "d930d022-36f9-4f16-9c29-f114847a9d3a" (UID: "d930d022-36f9-4f16-9c29-f114847a9d3a"). InnerVolumeSpecName "kube-api-access-zggfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:47.781895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.781870 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t"] Oct 30 13:10:47.782647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.782632 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d930d022-36f9-4f16-9c29-f114847a9d3a" containerName="revoke-keys" Oct 30 13:10:47.782706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.782649 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d930d022-36f9-4f16-9c29-f114847a9d3a" containerName="revoke-keys" Oct 30 13:10:47.782858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.782843 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d930d022-36f9-4f16-9c29-f114847a9d3a" containerName="revoke-keys" Oct 30 13:10:47.788087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.788064 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:47.816101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.815736 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t"] Oct 30 13:10:47.864962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.864929 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:47.873525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.873491 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:47.873673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.873571 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:47.873819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.873798 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrh9\" (UniqueName: \"kubernetes.io/projected/776c668a-bd43-44e1-99f3-ad9dea63a1a1-kube-api-access-vzrh9\") pod \"maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:47.873986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.873970 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d930d022-36f9-4f16-9c29-f114847a9d3a-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.874038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.873994 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zggfp\" (UniqueName: \"kubernetes.io/projected/d930d022-36f9-4f16-9c29-f114847a9d3a-kube-api-access-zggfp\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:47.975233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.975146 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrh9\" (UniqueName: \"kubernetes.io/projected/776c668a-bd43-44e1-99f3-ad9dea63a1a1-kube-api-access-vzrh9\") pod \"maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:47.975393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.975279 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:47.975393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.975308 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:47.975994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.975965 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:47.977835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.977808 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:47.997819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:47.997791 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrh9\" (UniqueName: \"kubernetes.io/projected/776c668a-bd43-44e1-99f3-ad9dea63a1a1-kube-api-access-vzrh9\") pod \"maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:48.117755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.117724 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg"] Oct 30 13:10:48.119605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.119580 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:48.143205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.143172 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5350dd6a-69ckg"] Oct 30 13:10:48.144388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.144311 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:48.333194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.333163 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:48.442577 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:10:48.442542 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776c668a_bd43_44e1_99f3_ad9dea63a1a1.slice/crio-e7bc5cf7b461cb0b6edd50676f4bba1b87b4106ee537fd42e9e8452b02834727 WatchSource:0}: Error finding container e7bc5cf7b461cb0b6edd50676f4bba1b87b4106ee537fd42e9e8452b02834727: Status 404 returned error can't find the container with id e7bc5cf7b461cb0b6edd50676f4bba1b87b4106ee537fd42e9e8452b02834727 Oct 30 13:10:48.444592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.444567 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t"] Oct 30 13:10:48.484001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.483971 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" event={"ID":"776c668a-bd43-44e1-99f3-ad9dea63a1a1","Type":"ContainerStarted","Data":"e7bc5cf7b461cb0b6edd50676f4bba1b87b4106ee537fd42e9e8452b02834727"} Oct 30 13:10:48.485776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.485748 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" event={"ID":"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3","Type":"ContainerStarted","Data":"90610a1926b3e5530d577ff99b7ee9b20ac702e8eac1904f540d24619d580113"} Oct 30 13:10:48.485898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.485794 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" event={"ID":"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3","Type":"ContainerStarted","Data":"1a3c9fd1dee341abf47f8b269f8eb2e288cecb319c0c7c0445dd6d786d9259a1"} Oct 30 13:10:48.487091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.487073 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" event={"ID":"d930d022-36f9-4f16-9c29-f114847a9d3a","Type":"ContainerDied","Data":"588c29fed55a674a2e58fcba0009913db1c86b26c6d9aeb3b96b79ede6b10e7f"} Oct 30 13:10:48.487174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.487103 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f" Oct 30 13:10:48.487214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.487105 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="588c29fed55a674a2e58fcba0009913db1c86b26c6d9aeb3b96b79ede6b10e7f" Oct 30 13:10:48.527264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.527207 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" podStartSLOduration=1.527189186 podStartE2EDuration="1.527189186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:48.52452151 +0000 UTC m=+1155.339715925" watchObservedRunningTime="2026-10-30 13:10:48.527189186 +0000 UTC m=+1155.342383611" Oct 30 13:10:48.844909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:48.844867 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:49.340076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:49.340041 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b780527-09c3-41ff-b3c6-58f78aae38ac" path="/var/lib/kubelet/pods/2b780527-09c3-41ff-b3c6-58f78aae38ac/volumes" Oct 30 13:10:49.497293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:49.497258 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" event={"ID":"776c668a-bd43-44e1-99f3-ad9dea63a1a1","Type":"ContainerStarted","Data":"9188c6cfa4de381431932c93162065cef4dda9167970942aca0f18e2975939aa"} Oct 30 13:10:49.499115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:49.499085 5049 generic.go:356] "Generic (PLEG): container finished" podID="1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3" containerID="90610a1926b3e5530d577ff99b7ee9b20ac702e8eac1904f540d24619d580113" exitCode=0 Oct 30 13:10:49.499238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:49.499147 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" event={"ID":"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3","Type":"ContainerDied","Data":"90610a1926b3e5530d577ff99b7ee9b20ac702e8eac1904f540d24619d580113"} Oct 30 13:10:49.539776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:49.539726 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" podStartSLOduration=2.539711915 podStartE2EDuration="2.539711915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:49.535046464 +0000 UTC m=+1156.350240890" watchObservedRunningTime="2026-10-30 13:10:49.539711915 +0000 UTC m=+1156.354906338" Oct 30 13:10:50.514103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.514065 5049 generic.go:356] "Generic (PLEG): container finished" podID="776c668a-bd43-44e1-99f3-ad9dea63a1a1" containerID="9188c6cfa4de381431932c93162065cef4dda9167970942aca0f18e2975939aa" exitCode=0 Oct 30 13:10:50.514634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.514149 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" event={"ID":"776c668a-bd43-44e1-99f3-ad9dea63a1a1","Type":"ContainerDied","Data":"9188c6cfa4de381431932c93162065cef4dda9167970942aca0f18e2975939aa"} Oct 30 13:10:50.725213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.725180 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:50.789212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.789139 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:50.843351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.840532 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-kube-api-access-hqmxh\" (UniqueName: \"kubernetes.io/projected/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-kube-api-access-hqmxh\") pod \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " Oct 30 13:10:50.843351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.840739 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-openshift-ca\") pod \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " Oct 30 13:10:50.843351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.840805 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-xks-ca\") pod \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\" (UID: \"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3\") " Oct 30 13:10:50.843351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.842648 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-openshift-ca" pod "1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3" (UID: "1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:50.846869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.846409 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-kube-api-access-hqmxh" pod "1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3" (UID: "1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3"). InnerVolumeSpecName "kube-api-access-hqmxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:50.865678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.865518 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-xks-ca" pod "1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3" (UID: "1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:50.943815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.943764 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqmxh\" (UniqueName: \"kubernetes.io/projected/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-kube-api-access-hqmxh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:50.943815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.943802 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:50.943815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.943818 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:50.944080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:50.943883 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:51.031994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:51.031960 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v"] Oct 30 13:10:51.037644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:51.037614 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c097f0d-8sm6v"] Oct 30 13:10:51.339392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:51.339359 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f5817b-e848-4027-9de6-8414a4cb3a91" path="/var/lib/kubelet/pods/d9f5817b-e848-4027-9de6-8414a4cb3a91/volumes" Oct 30 13:10:51.640723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:51.640631 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" Oct 30 13:10:51.644852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:51.643574 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc" event={"ID":"1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3","Type":"ContainerDied","Data":"1a3c9fd1dee341abf47f8b269f8eb2e288cecb319c0c7c0445dd6d786d9259a1"} Oct 30 13:10:51.644852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:51.643637 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a3c9fd1dee341abf47f8b269f8eb2e288cecb319c0c7c0445dd6d786d9259a1" Oct 30 13:10:51.997139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:51.997116 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:52.065228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.065186 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-openshift-ca\") pod \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " Oct 30 13:10:52.065445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.065363 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776c668a-bd43-44e1-99f3-ad9dea63a1a1-kube-api-access-vzrh9\" (UniqueName: \"kubernetes.io/projected/776c668a-bd43-44e1-99f3-ad9dea63a1a1-kube-api-access-vzrh9\") pod \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " Oct 30 13:10:52.065445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.065399 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-xks-ca\") pod \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\" (UID: \"776c668a-bd43-44e1-99f3-ad9dea63a1a1\") " Oct 30 13:10:52.066031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.066000 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-openshift-ca" pod "776c668a-bd43-44e1-99f3-ad9dea63a1a1" (UID: "776c668a-bd43-44e1-99f3-ad9dea63a1a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:52.066224 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.066205 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:52.068295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.068232 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-xks-ca" pod "776c668a-bd43-44e1-99f3-ad9dea63a1a1" (UID: "776c668a-bd43-44e1-99f3-ad9dea63a1a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:52.068295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.068257 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776c668a-bd43-44e1-99f3-ad9dea63a1a1-kube-api-access-vzrh9" pod "776c668a-bd43-44e1-99f3-ad9dea63a1a1" (UID: "776c668a-bd43-44e1-99f3-ad9dea63a1a1"). InnerVolumeSpecName "kube-api-access-vzrh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:52.167259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.167227 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrh9\" (UniqueName: \"kubernetes.io/projected/776c668a-bd43-44e1-99f3-ad9dea63a1a1-kube-api-access-vzrh9\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:52.167259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.167257 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c668a-bd43-44e1-99f3-ad9dea63a1a1-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:10:52.581048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.581003 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:52.659219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.658833 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" event={"ID":"776c668a-bd43-44e1-99f3-ad9dea63a1a1","Type":"ContainerDied","Data":"e7bc5cf7b461cb0b6edd50676f4bba1b87b4106ee537fd42e9e8452b02834727"} Oct 30 13:10:52.659219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.658897 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7bc5cf7b461cb0b6edd50676f4bba1b87b4106ee537fd42e9e8452b02834727" Oct 30 13:10:52.659219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:52.658970 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t" Oct 30 13:10:54.107347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.103145 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md"] Oct 30 13:10:54.112342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.110391 5049 cpu_manager.go:414] "containerMap: removing container" podUID="776c668a-bd43-44e1-99f3-ad9dea63a1a1" containerName="revoke-keys" Oct 30 13:10:54.112342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.110425 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776c668a-bd43-44e1-99f3-ad9dea63a1a1" containerName="revoke-keys" Oct 30 13:10:54.112342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.110459 5049 cpu_manager.go:414] "containerMap: removing container" podUID="1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3" containerName="revoke-keys" Oct 30 13:10:54.112342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.110468 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3" containerName="revoke-keys" Oct 30 13:10:54.112342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.110714 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="776c668a-bd43-44e1-99f3-ad9dea63a1a1" containerName="revoke-keys" Oct 30 13:10:54.112342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.110734 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3" containerName="revoke-keys" Oct 30 13:10:54.120352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.119941 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.203344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.198082 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.203344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.198137 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.203344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.198186 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.203344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.198253 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.203344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.198305 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkws\" (UniqueName: \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-kube-api-access-2qkws\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.203344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.198393 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.203344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.198447 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.268346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.261519 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.268346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.261589 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.421344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.412724 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.421344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.412781 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.421344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.412854 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.421344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.412950 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.421344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.413019 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkws\" (UniqueName: \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-kube-api-access-2qkws\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.421344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.413124 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.421344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.413193 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.421344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.413266 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.421344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.413312 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.421344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.415247 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.430841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.430804 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.457527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.457477 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.480349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.480211 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.483337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.481456 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:54.483337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.481625 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.516344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.510394 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:10:54.653471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.653416 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:54.735400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.732677 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:54.748345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.742708 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:54.748345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.742812 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:54.760359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.737387 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md"] Oct 30 13:10:54.878360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.876097 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkws\" (UniqueName: \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-kube-api-access-2qkws\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.929348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.901730 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.959370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.959210 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:54.990167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.990034 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:54.991462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:54.991095 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:55.001937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.001868 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:55.022031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.021471 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.098370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.091593 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded" Oct 30 13:10:55.098370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.096104 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:55.211190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.211150 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 13:10:55.257673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.256655 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.489606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.483617 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:10:55.489606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.484445 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:10:55.516158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.516036 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.517273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.517239 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:10:55.654449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.654403 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.663353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.655232 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:10:55.767879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.767786 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:10:55.768588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.768562 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:10:55.811346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.774315 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.909427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.815289 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:55.909427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.907216 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:10:55.955586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.955556 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:10:55.965702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.960578 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.974636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:55.973698 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:10:56.211980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:56.211935 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:56.242353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:56.224440 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:10:56.276610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:56.258501 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:10:56.277403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:56.277373 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:10:56.325345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:56.294413 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:56.525135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:56.471295 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:10:56.525135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:56.523790 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:10:56.535475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:56.535419 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:10:56.806034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:56.805994 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 13:10:56.807050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:56.806899 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:57.005647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.005446 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:57.021003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.020944 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:57.069910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.061714 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:10:57.069910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.062084 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:10:57.069910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.066424 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:57.069910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.066557 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:57.069910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.066649 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:57.069910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.067247 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:10:57.069910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.067314 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:10:57.083528 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:10:57.076895 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:10:57.100376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.098015 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:10:57.114695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.114669 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:10:57.144909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.141913 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7"] Oct 30 13:10:57.155990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.155962 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addee7da-5njc7"] Oct 30 13:10:57.203966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.203930 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md"] Oct 30 13:10:57.305299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.305051 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:57.351522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.351347 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dcc4b39-15f3-4c12-bc46-56c3ec7d4026" path="/var/lib/kubelet/pods/2dcc4b39-15f3-4c12-bc46-56c3ec7d4026/volumes" Oct 30 13:10:57.461611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.461578 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" event={"ID":"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610","Type":"ContainerStarted","Data":"62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95"} Oct 30 13:10:57.461772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.461640 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" event={"ID":"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610","Type":"ContainerStarted","Data":"5657fe155bb935345525e3cf07da91593e4288caaf51f4d0de222bcd53b75f8a"} Oct 30 13:10:57.480581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.480549 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:57.516461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.516398 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podStartSLOduration=3.516377998 podStartE2EDuration="3.516377998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:57.51147186 +0000 UTC m=+1164.326666282" watchObservedRunningTime="2026-10-30 13:10:57.516377998 +0000 UTC m=+1164.331572422" Oct 30 13:10:57.588370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.588334 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt"] Oct 30 13:10:57.777569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.777479 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:57.991629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:57.990194 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:58.098899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:58.098868 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:10:58.100200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:58.100171 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 13:10:58.100314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:58.100224 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 13:10:58.412399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:58.411753 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:58.649454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:58.649286 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:58.781160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:58.781065 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4"] Oct 30 13:10:58.840345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:58.838931 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:59.059412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:59.059372 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:59.125428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:59.123500 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 13:10:59.125428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:10:59.123586 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 13:11:00.099068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:00.099036 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 13:11:00.099528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:00.099090 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 13:11:00.744983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:00.744937 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:00.892345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:00.869105 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z"] Oct 30 13:11:00.986039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:00.985025 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:00.990364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:00.990096 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z"] Oct 30 13:11:01.004227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.004203 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:11:01.004380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.004302 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:11:01.082254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.082213 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-187291ac-hk42z\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:01.082467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.082449 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7ffb\" (UniqueName: \"kubernetes.io/projected/24995599-c47e-4abd-a57d-a589677fe6f1-kube-api-access-n7ffb\") pod \"maas-api-revoke-sub-187291ac-hk42z\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:01.082467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.082516 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-187291ac-hk42z\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:01.099631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.099601 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 13:11:01.099983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.099654 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 13:11:01.184332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.184275 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-187291ac-hk42z\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:01.184516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.184475 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7ffb\" (UniqueName: \"kubernetes.io/projected/24995599-c47e-4abd-a57d-a589677fe6f1-kube-api-access-n7ffb\") pod \"maas-api-revoke-sub-187291ac-hk42z\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:01.184579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.184560 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-187291ac-hk42z\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:01.185351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.185310 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-187291ac-hk42z\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:01.188249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.188223 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-187291ac-hk42z\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:01.254415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.238122 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7ffb\" (UniqueName: \"kubernetes.io/projected/24995599-c47e-4abd-a57d-a589677fe6f1-kube-api-access-n7ffb\") pod \"maas-api-revoke-sub-187291ac-hk42z\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:01.339530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.339386 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:01.371431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.371403 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:01.783056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.776993 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z"] Oct 30 13:11:01.831767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:01.831726 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:02.112118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:02.111012 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 13:11:02.112118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:02.111080 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 13:11:02.370476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:02.370384 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565"] Oct 30 13:11:02.400392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:02.400354 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:02.507351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:02.505505 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3466c89d-w4565"] Oct 30 13:11:02.589362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:02.589309 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" podUID="94ffd100-3d77-4848-8001-72320ba194d4" containerName="payload-processing" containerID="cri-o://5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4" gracePeriod=25 Oct 30 13:11:02.634099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:02.634010 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" event={"ID":"24995599-c47e-4abd-a57d-a589677fe6f1","Type":"ContainerStarted","Data":"0a80492538f7a67b8c3416b8803d379b328b9a4b058d2c888a9bf2a1828367a3"} Oct 30 13:11:02.829395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:02.817413 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:03.083786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.083720 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2"] Oct 30 13:11:03.094333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.094284 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ef410a-xg7w2"] Oct 30 13:11:03.104430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.104398 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 13:11:03.104695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.104662 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 13:11:03.294776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.294747 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:11:03.387504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.386398 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2acfffb1-ff93-4bb6-a30c-d27bbc1328b8" path="/var/lib/kubelet/pods/2acfffb1-ff93-4bb6-a30c-d27bbc1328b8/volumes" Oct 30 13:11:03.387504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.386925 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91b9bfc-9b4f-4a4a-bc3c-299c51529401" path="/var/lib/kubelet/pods/b91b9bfc-9b4f-4a4a-bc3c-299c51529401/volumes" Oct 30 13:11:03.489822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.489741 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ffd100-3d77-4848-8001-72320ba194d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ffd100-3d77-4848-8001-72320ba194d4-plugins-config-volume\") pod \"94ffd100-3d77-4848-8001-72320ba194d4\" (UID: \"94ffd100-3d77-4848-8001-72320ba194d4\") " Oct 30 13:11:03.490031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.490013 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ffd100-3d77-4848-8001-72320ba194d4-kube-api-access-sw4fv\" (UniqueName: \"kubernetes.io/projected/94ffd100-3d77-4848-8001-72320ba194d4-kube-api-access-sw4fv\") pod \"94ffd100-3d77-4848-8001-72320ba194d4\" (UID: \"94ffd100-3d77-4848-8001-72320ba194d4\") " Oct 30 13:11:03.490252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.490228 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ffd100-3d77-4848-8001-72320ba194d4-plugins-config-volume" pod "94ffd100-3d77-4848-8001-72320ba194d4" (UID: "94ffd100-3d77-4848-8001-72320ba194d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:03.491060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.491038 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ffd100-3d77-4848-8001-72320ba194d4-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.493125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.493101 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ffd100-3d77-4848-8001-72320ba194d4-kube-api-access-sw4fv" pod "94ffd100-3d77-4848-8001-72320ba194d4" (UID: "94ffd100-3d77-4848-8001-72320ba194d4"). InnerVolumeSpecName "kube-api-access-sw4fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:03.592793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.592761 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw4fv\" (UniqueName: \"kubernetes.io/projected/94ffd100-3d77-4848-8001-72320ba194d4-kube-api-access-sw4fv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.671450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.671413 5049 generic.go:356] "Generic (PLEG): container finished" podID="94ffd100-3d77-4848-8001-72320ba194d4" containerID="5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4" exitCode=0 Oct 30 13:11:03.671836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.671814 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" event={"ID":"94ffd100-3d77-4848-8001-72320ba194d4","Type":"ContainerDied","Data":"5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4"} Oct 30 13:11:03.671919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.671872 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" event={"ID":"94ffd100-3d77-4848-8001-72320ba194d4","Type":"ContainerDied","Data":"7d5009cd059df35ec031110f2d9a12d20f5ec86e3bc2cfd6c1cf2bcf3b45c7f9"} Oct 30 13:11:03.671919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.671902 5049 scope.go:122] "RemoveContainer" containerID="5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4" Oct 30 13:11:03.672065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.672051 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt" Oct 30 13:11:03.673776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.673749 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" event={"ID":"24995599-c47e-4abd-a57d-a589677fe6f1","Type":"ContainerStarted","Data":"0483cb4ddbeb6ddcf1a0e7521c6136e8422c06519e14c062ff794774cfdfe615"} Oct 30 13:11:03.762595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.762556 5049 scope.go:122] "RemoveContainer" containerID="5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4" Oct 30 13:11:03.763030 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:11:03.762968 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4\": container with ID starting with 5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4 not found: ID does not exist" containerID="5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4" Oct 30 13:11:03.763030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.763009 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4"} err="failed to get container status \"5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4\": rpc error: code = NotFound desc = could not find container \"5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4\": container with ID starting with 5b1904865c0e1a7ff3a06bc11ee959e3d0a70ef25fbfb68816d2b6b994ab5bf4 not found: ID does not exist" Oct 30 13:11:03.782063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.782031 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" podUID="d980ebfb-c1ec-491f-b6a7-4b14025d9027" containerName="payload-pre-processing" containerID="cri-o://ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954" gracePeriod=25 Oct 30 13:11:03.788728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.788699 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt"] Oct 30 13:11:03.794540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:03.794515 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b31c2926-58d9db84c8-wczlt"] Oct 30 13:11:04.099209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.099175 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 13:11:04.099510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.099480 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 13:11:04.258541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.258505 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:11:04.338396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.337393 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d980ebfb-c1ec-491f-b6a7-4b14025d9027-kube-api-access-lrr5h\" (UniqueName: \"kubernetes.io/projected/d980ebfb-c1ec-491f-b6a7-4b14025d9027-kube-api-access-lrr5h\") pod \"d980ebfb-c1ec-491f-b6a7-4b14025d9027\" (UID: \"d980ebfb-c1ec-491f-b6a7-4b14025d9027\") " Oct 30 13:11:04.338396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.337626 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d980ebfb-c1ec-491f-b6a7-4b14025d9027-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d980ebfb-c1ec-491f-b6a7-4b14025d9027-plugins-config-volume\") pod \"d980ebfb-c1ec-491f-b6a7-4b14025d9027\" (UID: \"d980ebfb-c1ec-491f-b6a7-4b14025d9027\") " Oct 30 13:11:04.338960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.338692 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d980ebfb-c1ec-491f-b6a7-4b14025d9027-plugins-config-volume" pod "d980ebfb-c1ec-491f-b6a7-4b14025d9027" (UID: "d980ebfb-c1ec-491f-b6a7-4b14025d9027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:04.366622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.366580 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d980ebfb-c1ec-491f-b6a7-4b14025d9027-kube-api-access-lrr5h" pod "d980ebfb-c1ec-491f-b6a7-4b14025d9027" (UID: "d980ebfb-c1ec-491f-b6a7-4b14025d9027"). InnerVolumeSpecName "kube-api-access-lrr5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:04.372072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.372033 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd"] Oct 30 13:11:04.372448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.372374 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" podUID="d694fe15-a212-46e5-941f-14b9f63da7a3" containerName="maas-api" containerID="cri-o://040a782035e0c59411a77613dbd6b4afc3e0c952404086c3c239d4a3a2aef0ea" gracePeriod=30 Oct 30 13:11:04.443879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.443842 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrr5h\" (UniqueName: \"kubernetes.io/projected/d980ebfb-c1ec-491f-b6a7-4b14025d9027-kube-api-access-lrr5h\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:04.444146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.444127 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d980ebfb-c1ec-491f-b6a7-4b14025d9027-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:04.450656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.447719 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:04.787392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.784217 5049 generic.go:356] "Generic (PLEG): container finished" podID="d980ebfb-c1ec-491f-b6a7-4b14025d9027" containerID="ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954" exitCode=0 Oct 30 13:11:04.787392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.784407 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" event={"ID":"d980ebfb-c1ec-491f-b6a7-4b14025d9027","Type":"ContainerDied","Data":"ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954"} Oct 30 13:11:04.787392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.784501 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" event={"ID":"d980ebfb-c1ec-491f-b6a7-4b14025d9027","Type":"ContainerDied","Data":"de02a9b98a46428388cd5deaf3e89bb9326ef5b3e597d7e99848cb1cc2fbb97f"} Oct 30 13:11:04.787392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.784547 5049 scope.go:122] "RemoveContainer" containerID="ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954" Oct 30 13:11:04.787392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.784789 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4" Oct 30 13:11:04.848514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.848475 5049 generic.go:356] "Generic (PLEG): container finished" podID="d694fe15-a212-46e5-941f-14b9f63da7a3" containerID="040a782035e0c59411a77613dbd6b4afc3e0c952404086c3c239d4a3a2aef0ea" exitCode=0 Oct 30 13:11:04.848935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.848906 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" event={"ID":"d694fe15-a212-46e5-941f-14b9f63da7a3","Type":"ContainerDied","Data":"040a782035e0c59411a77613dbd6b4afc3e0c952404086c3c239d4a3a2aef0ea"} Oct 30 13:11:04.915343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.913914 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:04.990345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.986938 5049 generic.go:356] "Generic (PLEG): container finished" podID="24995599-c47e-4abd-a57d-a589677fe6f1" containerID="0483cb4ddbeb6ddcf1a0e7521c6136e8422c06519e14c062ff794774cfdfe615" exitCode=0 Oct 30 13:11:04.990345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:04.987050 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" event={"ID":"24995599-c47e-4abd-a57d-a589677fe6f1","Type":"ContainerDied","Data":"0483cb4ddbeb6ddcf1a0e7521c6136e8422c06519e14c062ff794774cfdfe615"} Oct 30 13:11:05.127517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.127471 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:05.135771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.135676 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 13:11:05.136262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.136227 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 13:11:05.440259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.438469 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:05.628033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.622131 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:05.628033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.622241 5049 scope.go:122] "RemoveContainer" containerID="ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954" Oct 30 13:11:05.823055 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:11:05.823011 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954\": container with ID starting with ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954 not found: ID does not exist" containerID="ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954" Oct 30 13:11:05.823351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.823302 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954"} err="failed to get container status \"ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954\": rpc error: code = NotFound desc = could not find container \"ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954\": container with ID starting with ea7a7c0ff23f503147c0c1318166732a70f2499de2a6c43b0bb9216a3b24c954 not found: ID does not exist" Oct 30 13:11:05.869347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.867050 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:05.869347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.867184 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:05.968654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.968609 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:05.998874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:05.998253 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ffd100-3d77-4848-8001-72320ba194d4" path="/var/lib/kubelet/pods/94ffd100-3d77-4848-8001-72320ba194d4/volumes" Oct 30 13:11:06.060420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.045368 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:06.094845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.071464 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:06.122351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.103871 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:06.218444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.217117 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4"] Oct 30 13:11:06.282680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.282646 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:06.406434 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.405555 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:11:06.485591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.485553 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7e2137bc-9dfbdb4ff-fdlj4"] Oct 30 13:11:06.519750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.519631 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:06.542529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.542484 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:06.568686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.568651 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-maas-api-tls\") pod \"d694fe15-a212-46e5-941f-14b9f63da7a3\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " Oct 30 13:11:06.573651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.568984 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d694fe15-a212-46e5-941f-14b9f63da7a3-kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/d694fe15-a212-46e5-941f-14b9f63da7a3-kube-api-access-gxwxt\") pod \"d694fe15-a212-46e5-941f-14b9f63da7a3\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " Oct 30 13:11:06.574197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.574168 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-metrics-certs\") pod \"d694fe15-a212-46e5-941f-14b9f63da7a3\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " Oct 30 13:11:06.599636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.598646 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d694fe15-a212-46e5-941f-14b9f63da7a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d694fe15-a212-46e5-941f-14b9f63da7a3-openshift-service-ca\") pod \"d694fe15-a212-46e5-941f-14b9f63da7a3\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " Oct 30 13:11:06.601874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.580400 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:06.614745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.611516 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d694fe15-a212-46e5-941f-14b9f63da7a3-openshift-service-ca" pod "d694fe15-a212-46e5-941f-14b9f63da7a3" (UID: "d694fe15-a212-46e5-941f-14b9f63da7a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:06.615503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.615474 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-metrics-certs" pod "d694fe15-a212-46e5-941f-14b9f63da7a3" (UID: "d694fe15-a212-46e5-941f-14b9f63da7a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:06.624405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.617792 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-maas-api-tls" pod "d694fe15-a212-46e5-941f-14b9f63da7a3" (UID: "d694fe15-a212-46e5-941f-14b9f63da7a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:06.722043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.721996 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d694fe15-a212-46e5-941f-14b9f63da7a3-kube-api-access-gxwxt" pod "d694fe15-a212-46e5-941f-14b9f63da7a3" (UID: "d694fe15-a212-46e5-941f-14b9f63da7a3"). InnerVolumeSpecName "kube-api-access-gxwxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:06.735894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.735842 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d694fe15-a212-46e5-941f-14b9f63da7a3-kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/d694fe15-a212-46e5-941f-14b9f63da7a3-kube-api-access-gxwxt\") pod \"d694fe15-a212-46e5-941f-14b9f63da7a3\" (UID: \"d694fe15-a212-46e5-941f-14b9f63da7a3\") " Oct 30 13:11:06.762493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.745514 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d694fe15-a212-46e5-941f-14b9f63da7a3-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:06.764104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.764071 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:06.764294 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.764280 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d694fe15-a212-46e5-941f-14b9f63da7a3-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:06.764413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.763023 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:06.764726 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:11:06.764701 5049 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/d694fe15-a212-46e5-941f-14b9f63da7a3/volumes/kubernetes.io~projected/kube-api-access-gxwxt Oct 30 13:11:06.764821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.764777 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d694fe15-a212-46e5-941f-14b9f63da7a3-kube-api-access-gxwxt" pod "d694fe15-a212-46e5-941f-14b9f63da7a3" (UID: "d694fe15-a212-46e5-941f-14b9f63da7a3"). InnerVolumeSpecName "kube-api-access-gxwxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:06.856101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.815829 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:06.875351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.872688 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/d694fe15-a212-46e5-941f-14b9f63da7a3-kube-api-access-gxwxt\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:06.948275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.948241 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc"] Oct 30 13:11:06.985209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.985168 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:06.989669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.985472 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b31c2926-5qpgc"] Oct 30 13:11:06.990151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:06.990121 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:11:07.082347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.069307 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:07.181220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.070382 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t"] Oct 30 13:11:07.181220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.167459 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:11:07.181220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.167537 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:07.306510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.248027 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e2137bc-7rp8t"] Oct 30 13:11:07.306510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.248093 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v"] Oct 30 13:11:07.306510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.248315 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" podUID="53322eb2-900b-48e6-8be4-fb7d1a655f22" containerName="istio-proxy" containerID="cri-o://76ac6b5007e3e0c39bb1441c335addc6a4387fb361a2fc3b92b6912a9a7bc010" gracePeriod=30 Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.318379 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl"] Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.319464 5049 cpu_manager.go:414] "containerMap: removing container" podUID="94ffd100-3d77-4848-8001-72320ba194d4" containerName="payload-processing" Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.319483 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ffd100-3d77-4848-8001-72320ba194d4" containerName="payload-processing" Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.319505 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d980ebfb-c1ec-491f-b6a7-4b14025d9027" containerName="payload-pre-processing" Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.319514 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d980ebfb-c1ec-491f-b6a7-4b14025d9027" containerName="payload-pre-processing" Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.319586 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d694fe15-a212-46e5-941f-14b9f63da7a3" containerName="maas-api" Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.319595 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d694fe15-a212-46e5-941f-14b9f63da7a3" containerName="maas-api" Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.319811 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ffd100-3d77-4848-8001-72320ba194d4" containerName="payload-processing" Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.319822 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d980ebfb-c1ec-491f-b6a7-4b14025d9027" containerName="payload-pre-processing" Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.319837 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d694fe15-a212-46e5-941f-14b9f63da7a3" containerName="maas-api" Oct 30 13:11:07.336637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.324155 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:07.369441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.355889 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl"] Oct 30 13:11:07.369441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.359126 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.384341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.377764 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-911fe1d6-openshift-default-dockercfg-gxzjx\"" Oct 30 13:11:07.401342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.399830 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3" path="/var/lib/kubelet/pods/1e90e5c8-ee4d-4ae5-9f2e-4be771ef52d3/volumes" Oct 30 13:11:07.421353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.413280 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776c668a-bd43-44e1-99f3-ad9dea63a1a1" path="/var/lib/kubelet/pods/776c668a-bd43-44e1-99f3-ad9dea63a1a1/volumes" Oct 30 13:11:07.421353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.414157 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d980ebfb-c1ec-491f-b6a7-4b14025d9027" path="/var/lib/kubelet/pods/d980ebfb-c1ec-491f-b6a7-4b14025d9027/volumes" Oct 30 13:11:07.453340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.448439 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z"] Oct 30 13:11:07.477052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.477019 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a334383-z8p8z"] Oct 30 13:11:07.480450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.479538 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:11:07.489955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.488399 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istiod-ca-cert\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.489955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.488461 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-socket\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.489955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.488574 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-token\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.489955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.488633 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plf8l\" (UniqueName: \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-kube-api-access-plf8l\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.489955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.488683 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-credential-socket\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.489955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.488760 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-envoy\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.489955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.488797 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-data\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.489955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.488952 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-certs\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.489955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.489123 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-podinfo\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.522439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.522298 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" event={"ID":"d694fe15-a212-46e5-941f-14b9f63da7a3","Type":"ContainerDied","Data":"1e4c5dc2a5af4280d0a6df646322f3464b8369bde5df68e21de7ade7be652092"} Oct 30 13:11:07.522701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.522682 5049 scope.go:122] "RemoveContainer" containerID="040a782035e0c59411a77613dbd6b4afc3e0c952404086c3c239d4a3a2aef0ea" Oct 30 13:11:07.524763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.522929 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:07.524970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.524435 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd" Oct 30 13:11:07.564053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.564009 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" event={"ID":"24995599-c47e-4abd-a57d-a589677fe6f1","Type":"ContainerDied","Data":"0a80492538f7a67b8c3416b8803d379b328b9a4b058d2c888a9bf2a1828367a3"} Oct 30 13:11:07.564211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.564128 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a80492538f7a67b8c3416b8803d379b328b9a4b058d2c888a9bf2a1828367a3" Oct 30 13:11:07.590109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.590060 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-openshift-ca\") pod \"24995599-c47e-4abd-a57d-a589677fe6f1\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " Oct 30 13:11:07.590244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.590204 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24995599-c47e-4abd-a57d-a589677fe6f1-kube-api-access-n7ffb\" (UniqueName: \"kubernetes.io/projected/24995599-c47e-4abd-a57d-a589677fe6f1-kube-api-access-n7ffb\") pod \"24995599-c47e-4abd-a57d-a589677fe6f1\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " Oct 30 13:11:07.590915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.590887 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-openshift-ca" pod "24995599-c47e-4abd-a57d-a589677fe6f1" (UID: "24995599-c47e-4abd-a57d-a589677fe6f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:07.596855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.596824 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24995599-c47e-4abd-a57d-a589677fe6f1-kube-api-access-n7ffb" pod "24995599-c47e-4abd-a57d-a589677fe6f1" (UID: "24995599-c47e-4abd-a57d-a589677fe6f1"). InnerVolumeSpecName "kube-api-access-n7ffb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:07.599513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.599489 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-xks-ca\") pod \"24995599-c47e-4abd-a57d-a589677fe6f1\" (UID: \"24995599-c47e-4abd-a57d-a589677fe6f1\") " Oct 30 13:11:07.599924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.599903 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istiod-ca-cert\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.600000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.599943 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-socket\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.600059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.600040 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-token\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.600118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.600097 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plf8l\" (UniqueName: \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-kube-api-access-plf8l\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.600176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.600159 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-credential-socket\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.600259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.600245 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-envoy\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.600314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.600285 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-data\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.600465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.600447 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-certs\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.600614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.600599 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-podinfo\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.600804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.600772 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:07.600804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.600797 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7ffb\" (UniqueName: \"kubernetes.io/projected/24995599-c47e-4abd-a57d-a589677fe6f1-kube-api-access-n7ffb\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:07.603588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.603556 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-xks-ca" pod "24995599-c47e-4abd-a57d-a589677fe6f1" (UID: "24995599-c47e-4abd-a57d-a589677fe6f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:07.628346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.604167 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-credential-socket\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.628346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.604375 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istiod-ca-cert\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.628346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.604634 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-certs\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.628346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.604679 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-data\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.628346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.605062 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-podinfo\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.628346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.605439 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-socket\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.628346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.616586 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-envoy\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.637925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.637380 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-token\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.683341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.647218 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plf8l\" (UniqueName: \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-kube-api-access-plf8l\") pod \"e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.716342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.713507 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24995599-c47e-4abd-a57d-a589677fe6f1-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:07.786681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.779380 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:11:07.786681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.781931 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:11:07.786681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.782975 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:11:07.822007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.818362 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9"] Oct 30 13:11:07.823786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.823753 5049 cpu_manager.go:414] "containerMap: removing container" podUID="24995599-c47e-4abd-a57d-a589677fe6f1" containerName="revoke-keys" Oct 30 13:11:07.824034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.823999 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24995599-c47e-4abd-a57d-a589677fe6f1" containerName="revoke-keys" Oct 30 13:11:07.825142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.825124 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="24995599-c47e-4abd-a57d-a589677fe6f1" containerName="revoke-keys" Oct 30 13:11:07.854386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.854351 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:07.858847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.858049 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:07.863087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.863025 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd"] Oct 30 13:11:07.905615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.905586 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b31c2926-6c6d8d88bb-mblgd"] Oct 30 13:11:07.906845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.906820 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-45c4d25f-openshift-default-dockercfg-pn6z7\"" Oct 30 13:11:07.911469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.911442 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9"] Oct 30 13:11:07.962752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.958456 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-data\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:07.962752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.958630 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b711a4-1edf-4405-b7cc-b493017f2949-istio-podinfo\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:07.962752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.958674 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-istio-token\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:07.962752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.958728 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-certs\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:07.962752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.958806 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-socket\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:07.962752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.958917 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-envoy\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:07.962752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.958951 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b711a4-1edf-4405-b7cc-b493017f2949-istiod-ca-cert\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:07.962752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.958980 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wh6\" (UniqueName: \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-kube-api-access-j2wh6\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:07.962752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.959028 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-credential-socket\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:07.973280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:07.973094 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:08.060461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.060273 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b711a4-1edf-4405-b7cc-b493017f2949-istio-podinfo\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.069135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.064639 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-istio-token\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.069135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.064721 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-certs\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.069135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.064785 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-socket\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.069135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.064915 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-envoy\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.069135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.064947 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b711a4-1edf-4405-b7cc-b493017f2949-istiod-ca-cert\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.069135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.064982 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wh6\" (UniqueName: \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-kube-api-access-j2wh6\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.069135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.065035 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-credential-socket\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.096568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.095632 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-data\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.096568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.066405 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-certs\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.096568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.066676 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-socket\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.096568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.067672 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b711a4-1edf-4405-b7cc-b493017f2949-istiod-ca-cert\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.096568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.069076 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b711a4-1edf-4405-b7cc-b493017f2949-istio-podinfo\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.096568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.065765 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-credential-socket\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.096568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.096452 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-data\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.224923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.224791 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:08.232384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.232299 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-envoy\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.234342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.234266 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-istio-token\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.237676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.237585 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wh6\" (UniqueName: \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-kube-api-access-j2wh6\") pod \"e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.456798 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:11:08.456760 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a94ef0_413f_48c1_aeee_2538cb0afb2f.slice/crio-5b18419f7fadfa67edbc3cb93be4d38998aeb440ec0ef7691f716f242c0be788 WatchSource:0}: Error finding container 5b18419f7fadfa67edbc3cb93be4d38998aeb440ec0ef7691f716f242c0be788: Status 404 returned error can't find the container with id 5b18419f7fadfa67edbc3cb93be4d38998aeb440ec0ef7691f716f242c0be788 Oct 30 13:11:08.459993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.459950 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:08.460110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.460044 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:08.460173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.460132 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:08.460541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.460514 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl"] Oct 30 13:11:08.517622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.517593 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:08.582912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.581561 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" event={"ID":"d5a94ef0-413f-48c1-aeee-2538cb0afb2f","Type":"ContainerStarted","Data":"66893a8d4e9570f84176c8883c8d1749c54a0c75bd02c94f6496e50bc8b92523"} Oct 30 13:11:08.582912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.581631 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" event={"ID":"d5a94ef0-413f-48c1-aeee-2538cb0afb2f","Type":"ContainerStarted","Data":"5b18419f7fadfa67edbc3cb93be4d38998aeb440ec0ef7691f716f242c0be788"} Oct 30 13:11:08.587342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.585697 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z" Oct 30 13:11:08.593140 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.593074 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:08.650145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.650081 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" podStartSLOduration=2.650061952 podStartE2EDuration="2.650061952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:08.644588552 +0000 UTC m=+1175.459782976" watchObservedRunningTime="2026-10-30 13:11:08.650061952 +0000 UTC m=+1175.465256377" Oct 30 13:11:08.804429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.804270 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:08.874844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:08.874672 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:09.103359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:09.103291 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9"] Oct 30 13:11:09.179176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:09.121568 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:09.179176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:09.121667 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:09.179176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:09.121713 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:09.343677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:09.343646 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c410dc-efe9-4661-aa8d-bc27ddd62c16" path="/var/lib/kubelet/pods/46c410dc-efe9-4661-aa8d-bc27ddd62c16/volumes" Oct 30 13:11:09.344313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:09.344295 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d694fe15-a212-46e5-941f-14b9f63da7a3" path="/var/lib/kubelet/pods/d694fe15-a212-46e5-941f-14b9f63da7a3/volumes" Oct 30 13:11:09.613838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:09.613801 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" event={"ID":"89b711a4-1edf-4405-b7cc-b493017f2949","Type":"ContainerStarted","Data":"56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff"} Oct 30 13:11:09.614417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:09.614389 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" event={"ID":"89b711a4-1edf-4405-b7cc-b493017f2949","Type":"ContainerStarted","Data":"dcf0147c92d34e9b7855172855f5439eb6aa5efe6d40981eee057b7075d09cc9"} Oct 30 13:11:09.664052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:09.663980 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" podStartSLOduration=2.6639582280000003 podStartE2EDuration="2.663958228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:09.655196037 +0000 UTC m=+1176.470390460" watchObservedRunningTime="2026-10-30 13:11:09.663958228 +0000 UTC m=+1176.479152652" Oct 30 13:11:09.865920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:09.865830 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:10.441855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:10.441816 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:10.519152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:10.519110 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:10.526758 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:10.526726 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:10.627363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:10.627293 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:10.627915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:10.627895 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:10.661431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:10.649426 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:11:10.661431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:10.650432 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:10.692402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:10.688597 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:11:10.841341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:10.834735 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:11.388612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:11.388578 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:11.570837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:11.570802 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:12.470652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:12.470615 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:12.673163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:12.673083 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:12.832529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:12.832481 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:13.690095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:13.690063 5049 generic.go:356] "Generic (PLEG): container finished" podID="53322eb2-900b-48e6-8be4-fb7d1a655f22" containerID="76ac6b5007e3e0c39bb1441c335addc6a4387fb361a2fc3b92b6912a9a7bc010" exitCode=0 Oct 30 13:11:13.690541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:13.690136 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" event={"ID":"53322eb2-900b-48e6-8be4-fb7d1a655f22","Type":"ContainerDied","Data":"76ac6b5007e3e0c39bb1441c335addc6a4387fb361a2fc3b92b6912a9a7bc010"} Oct 30 13:11:13.980031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:13.980000 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:11:14.052073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.052027 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-kube-api-access-5vq8x\" (UniqueName: \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-kube-api-access-5vq8x\") pod \"53322eb2-900b-48e6-8be4-fb7d1a655f22\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " Oct 30 13:11:14.052309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.052240 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-socket\") pod \"53322eb2-900b-48e6-8be4-fb7d1a655f22\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " Oct 30 13:11:14.052309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.052290 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-certs\") pod \"53322eb2-900b-48e6-8be4-fb7d1a655f22\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " Oct 30 13:11:14.052509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.052375 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53322eb2-900b-48e6-8be4-fb7d1a655f22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53322eb2-900b-48e6-8be4-fb7d1a655f22-istiod-ca-cert\") pod \"53322eb2-900b-48e6-8be4-fb7d1a655f22\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " Oct 30 13:11:14.052509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.052425 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-data\") pod \"53322eb2-900b-48e6-8be4-fb7d1a655f22\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " Oct 30 13:11:14.052608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.052511 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-credential-socket\") pod \"53322eb2-900b-48e6-8be4-fb7d1a655f22\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " Oct 30 13:11:14.052608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.052577 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-podinfo\") pod \"53322eb2-900b-48e6-8be4-fb7d1a655f22\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " Oct 30 13:11:14.052672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.052657 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-token\" (UniqueName: \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-token\") pod \"53322eb2-900b-48e6-8be4-fb7d1a655f22\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " Oct 30 13:11:14.053473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.053084 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-certs" pod "53322eb2-900b-48e6-8be4-fb7d1a655f22" (UID: "53322eb2-900b-48e6-8be4-fb7d1a655f22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:14.053473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.053173 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-data" pod "53322eb2-900b-48e6-8be4-fb7d1a655f22" (UID: "53322eb2-900b-48e6-8be4-fb7d1a655f22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:14.053473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.053198 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53322eb2-900b-48e6-8be4-fb7d1a655f22-istiod-ca-cert" pod "53322eb2-900b-48e6-8be4-fb7d1a655f22" (UID: "53322eb2-900b-48e6-8be4-fb7d1a655f22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:14.053473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.053333 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-credential-socket" pod "53322eb2-900b-48e6-8be4-fb7d1a655f22" (UID: "53322eb2-900b-48e6-8be4-fb7d1a655f22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:14.053473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.053392 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-envoy\") pod \"53322eb2-900b-48e6-8be4-fb7d1a655f22\" (UID: \"53322eb2-900b-48e6-8be4-fb7d1a655f22\") " Oct 30 13:11:14.053473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.053417 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-socket" pod "53322eb2-900b-48e6-8be4-fb7d1a655f22" (UID: "53322eb2-900b-48e6-8be4-fb7d1a655f22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:14.054132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.054107 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:14.054212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.054137 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:14.054212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.054152 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53322eb2-900b-48e6-8be4-fb7d1a655f22-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:14.054212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.054166 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:14.054212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.054179 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:14.056059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.056027 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-token" pod "53322eb2-900b-48e6-8be4-fb7d1a655f22" (UID: "53322eb2-900b-48e6-8be4-fb7d1a655f22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:14.057668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.057029 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-envoy" pod "53322eb2-900b-48e6-8be4-fb7d1a655f22" (UID: "53322eb2-900b-48e6-8be4-fb7d1a655f22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:14.058213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.058182 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-kube-api-access-5vq8x" pod "53322eb2-900b-48e6-8be4-fb7d1a655f22" (UID: "53322eb2-900b-48e6-8be4-fb7d1a655f22"). InnerVolumeSpecName "kube-api-access-5vq8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:14.060414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.060387 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-podinfo" pod "53322eb2-900b-48e6-8be4-fb7d1a655f22" (UID: "53322eb2-900b-48e6-8be4-fb7d1a655f22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:14.116314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.116280 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:14.155725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.155684 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:14.155725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.155720 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vq8x\" (UniqueName: \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-kube-api-access-5vq8x\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:14.155725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.155734 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:14.155959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.155746 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53322eb2-900b-48e6-8be4-fb7d1a655f22-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:11:14.270579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.270474 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:14.663346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.657898 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:14.712435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.710812 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" Oct 30 13:11:14.712435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.710823 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v" event={"ID":"53322eb2-900b-48e6-8be4-fb7d1a655f22","Type":"ContainerDied","Data":"82a79f59b05e5fdf60ba21b3c327c2f28c7a5ff119c251b746191339727e0652"} Oct 30 13:11:14.712435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.710892 5049 scope.go:122] "RemoveContainer" containerID="76ac6b5007e3e0c39bb1441c335addc6a4387fb361a2fc3b92b6912a9a7bc010" Oct 30 13:11:14.783287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.781779 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v"] Oct 30 13:11:14.793757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:14.793726 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7e2137bc-gw-openshift-default-55cd57565c-4cz8v"] Oct 30 13:11:15.348055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:15.348016 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53322eb2-900b-48e6-8be4-fb7d1a655f22" path="/var/lib/kubelet/pods/53322eb2-900b-48e6-8be4-fb7d1a655f22/volumes" Oct 30 13:11:16.031011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:16.030974 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9"] Oct 30 13:11:16.041567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:16.041537 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63da8efd-drpx9"] Oct 30 13:11:16.384779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:16.384743 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:17.337777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:17.337747 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d165b957-19eb-4710-998c-7bfa4cf683b8" path="/var/lib/kubelet/pods/d165b957-19eb-4710-998c-7bfa4cf683b8/volumes" Oct 30 13:11:18.063344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:18.058826 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:19.165368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:19.159166 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:19.485520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:19.485408 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:19.505452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:19.505403 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:11:19.538335 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:19.538278 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:19.820455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:19.820406 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.024469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.024418 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:11:20.025991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.025962 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:11:20.073546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.073417 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.074203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.074168 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:11:20.116461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.116412 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.216345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.216108 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.486402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.485989 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.510345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.486631 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:11:20.510345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.506098 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:11:20.510345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.507712 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:11:20.561142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.549787 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.561142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.550441 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:11:20.935343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.919970 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.982359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:20.949166 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:11:21.050989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.032501 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:21.050989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.046406 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:21.072732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.052133 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:11:21.072732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.052210 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:11:21.094099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.094042 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:11:21.099354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.099020 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 13:11:21.127870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.127834 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:11:21.211401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.129033 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:11:21.461342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.448520 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:11:21.569344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.568649 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:11:21.615835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.615792 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:11:21.716945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.668981 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:11:21.769728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.745407 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:21.871396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.841660 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:21.980394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:21.929234 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:11:22.073343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.048917 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:22.248401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.096849 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:22.248401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.109004 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:22.248401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.153692 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:22.454356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.450724 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:11:22.454356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.450883 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:11:22.454356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.450926 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:11:22.454356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.450968 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:11:22.454356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.451012 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:11:22.454356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.451060 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:11:22.454356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.451093 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:11:22.500340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.500225 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:11:22.519340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:22.503132 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:11:23.607816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:23.607762 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:23.774620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:23.774587 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:23.964831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:23.964743 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:25.587068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:25.585378 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:25.861592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:25.861516 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:26.678657 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:11:26.678624 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:11:27.372934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:27.372884 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:29.200533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.200502 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5"] Oct 30 13:11:29.201227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.201211 5049 cpu_manager.go:414] "containerMap: removing container" podUID="53322eb2-900b-48e6-8be4-fb7d1a655f22" containerName="istio-proxy" Oct 30 13:11:29.201227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.201227 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53322eb2-900b-48e6-8be4-fb7d1a655f22" containerName="istio-proxy" Oct 30 13:11:29.201404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.201393 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="53322eb2-900b-48e6-8be4-fb7d1a655f22" containerName="istio-proxy" Oct 30 13:11:29.205616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.205586 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.231679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.231650 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-911fe1d6-serving-cert\"" Oct 30 13:11:29.243759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.243732 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5"] Oct 30 13:11:29.335016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.334976 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:29.341316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.341282 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/471b846a-a62e-49fc-bb51-248d92dcc932-openshift-service-ca\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.341481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.341381 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-metrics-certs\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.341481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.341418 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-maas-api-tls\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.341632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.341605 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm2sc\" (UniqueName: \"kubernetes.io/projected/471b846a-a62e-49fc-bb51-248d92dcc932-kube-api-access-rm2sc\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.450481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.447492 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/471b846a-a62e-49fc-bb51-248d92dcc932-openshift-service-ca\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.450481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.447558 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-metrics-certs\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.450481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.447704 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-maas-api-tls\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.450481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.448001 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm2sc\" (UniqueName: \"kubernetes.io/projected/471b846a-a62e-49fc-bb51-248d92dcc932-kube-api-access-rm2sc\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.455127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.455036 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/471b846a-a62e-49fc-bb51-248d92dcc932-openshift-service-ca\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.478706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.478669 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-metrics-certs\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.488921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.488887 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-maas-api-tls\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.503354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.489829 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm2sc\" (UniqueName: \"kubernetes.io/projected/471b846a-a62e-49fc-bb51-248d92dcc932-kube-api-access-rm2sc\") pod \"maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.525620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.525586 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:29.867063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:29.867033 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5"] Oct 30 13:11:29.868014 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:11:29.867986 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471b846a_a62e_49fc_bb51_248d92dcc932.slice/crio-784b62d82c535a8fbd6e488bfa9227d8c9a3f8d67db32c67a0f6e8b1f0438ef5 WatchSource:0}: Error finding container 784b62d82c535a8fbd6e488bfa9227d8c9a3f8d67db32c67a0f6e8b1f0438ef5: Status 404 returned error can't find the container with id 784b62d82c535a8fbd6e488bfa9227d8c9a3f8d67db32c67a0f6e8b1f0438ef5 Oct 30 13:11:30.639347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:30.634570 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:30.639347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:30.636588 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" event={"ID":"471b846a-a62e-49fc-bb51-248d92dcc932","Type":"ContainerStarted","Data":"e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d"} Oct 30 13:11:30.639347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:30.636649 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" event={"ID":"471b846a-a62e-49fc-bb51-248d92dcc932","Type":"ContainerStarted","Data":"784b62d82c535a8fbd6e488bfa9227d8c9a3f8d67db32c67a0f6e8b1f0438ef5"} Oct 30 13:11:30.639347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:30.638003 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:30.688365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:30.688263 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" podStartSLOduration=1.3148775 podStartE2EDuration="1.688244324s" totalImagesPullingTime="373.366824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:30.682911814 +0000 UTC m=+1197.498106253" watchObservedRunningTime="2026-10-30 13:11:30.688244324 +0000 UTC m=+1197.503438761" Oct 30 13:11:31.247676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:31.247635 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:31.426737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:31.416688 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:31.570120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:31.570085 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:33.119486 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:33.119446 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:33.498364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:33.498236 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:11:33.501664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:33.501641 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:11:33.823299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:33.823266 5049 scope.go:122] "RemoveContainer" containerID="f213f1526dd1f8da8995e625ce3a3a75745d67338a26a37690e070f98b28eeae" Oct 30 13:11:33.835865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:33.835843 5049 scope.go:122] "RemoveContainer" containerID="cf7c03a72fb8758df70107ecd8a9d4aeb8b2829b4ea84bdfc38565a4035405bb" Oct 30 13:11:33.846828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:33.846803 5049 scope.go:122] "RemoveContainer" containerID="d363d56ac9bd33377bd2f213f410dfb36bcaea11744554ad8a4ed78334547afc" Oct 30 13:11:35.008353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.005122 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:35.074588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.074553 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr"] Oct 30 13:11:35.082265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.082230 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.120021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.119984 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr"] Oct 30 13:11:35.273488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.273403 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.273656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.273490 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4mw7\" (UniqueName: \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-kube-api-access-w4mw7\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.273656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.273535 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.273656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.273595 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.273780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.273740 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.273883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.273865 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.273920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.273898 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.273974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.273935 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.274077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.274059 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.375241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.375209 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.375241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.375248 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.375497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.375276 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.375497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.375359 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.375497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.375413 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.375497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.375432 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4mw7\" (UniqueName: \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-kube-api-access-w4mw7\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.375497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.375461 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.375781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.375550 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.375781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.375648 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.375781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.375771 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.376106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.376078 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.376196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.376105 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.376196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.376135 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.376196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.376149 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.377633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.377608 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.377888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.377867 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.397497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.397470 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.397625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.397607 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4mw7\" (UniqueName: \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-kube-api-access-w4mw7\") pod \"maas-default-gateway-openshift-default-64d4cc998c-4zhvr\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.405865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.405842 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:35.740720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.740695 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr"] Oct 30 13:11:35.741809 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:11:35.741780 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7ae366_ee2c_4d84_a62b_2e37405fbcf5.slice/crio-29d830b31d00088404d7478d705a0641025a77039b8137191273ef7b7083722f WatchSource:0}: Error finding container 29d830b31d00088404d7478d705a0641025a77039b8137191273ef7b7083722f: Status 404 returned error can't find the container with id 29d830b31d00088404d7478d705a0641025a77039b8137191273ef7b7083722f Oct 30 13:11:35.744658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.744624 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:35.744725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.744710 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:35.744768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:35.744746 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:36.741481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:36.741379 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" event={"ID":"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5","Type":"ContainerStarted","Data":"b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3"} Oct 30 13:11:36.741481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:36.741440 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" event={"ID":"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5","Type":"ContainerStarted","Data":"29d830b31d00088404d7478d705a0641025a77039b8137191273ef7b7083722f"} Oct 30 13:11:36.789823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:36.789730 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" podStartSLOduration=1.789711857 podStartE2EDuration="1.789711857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:36.786405191 +0000 UTC m=+1203.601599607" watchObservedRunningTime="2026-10-30 13:11:36.789711857 +0000 UTC m=+1203.604906282" Oct 30 13:11:36.897039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:36.896993 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:37.413011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:37.412976 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:37.413364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:37.413341 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-4zhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:37.425460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:37.425413 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 13:11:37.686911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:37.686836 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:11:38.420811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:38.420771 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-4zhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:38.421409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:38.421374 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 13:11:38.604731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:38.604692 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:38.847960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:38.847923 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:39.020817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:39.020778 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:39.419347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:39.417338 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-4zhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:39.419347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:39.417395 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 13:11:39.736993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:39.736901 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:40.422347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:40.414664 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-4zhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:40.422347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:40.414724 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 13:11:40.445260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:40.445221 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:41.418701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:41.418666 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-4zhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:41.419292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:41.419262 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 13:11:42.177456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:42.173931 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:42.406137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:42.406106 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-4zhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:42.406415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:42.406384 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 13:11:43.406728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:43.406680 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-4zhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:43.407224 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:43.406753 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 13:11:44.211097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:44.210890 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:44.470537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:44.470430 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:44.471055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:44.471033 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:44.472077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:44.472054 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:11:46.078343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:46.072660 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:46.781550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:46.781470 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:47.538848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:47.538812 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:47.803885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:47.803851 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b"] Oct 30 13:11:47.812706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:47.812678 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:47.840681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:47.840652 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5a6281-openshift-default-dockercfg-k6zpv\"" Oct 30 13:11:47.850399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:47.850370 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b"] Oct 30 13:11:48.035351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:47.992834 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:48.037299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.037265 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-certs\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.038788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.038756 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqwf\" (UniqueName: \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-kube-api-access-8rqwf\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.039273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.039256 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-token\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.040845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.040819 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-data\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.043882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.043859 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-envoy\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.044891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.044871 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istiod-ca-cert\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.045679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.045659 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-podinfo\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.046058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.045864 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-credential-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.046209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.046193 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.169397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.168775 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-certs\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.169397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.168816 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqwf\" (UniqueName: \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-kube-api-access-8rqwf\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.169397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.168863 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-token\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.169397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.168902 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-data\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.169397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.168922 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-envoy\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.169397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.168958 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istiod-ca-cert\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.169397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.169042 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-podinfo\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.169397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.169070 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-credential-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.169397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.169087 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.170009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.169924 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-certs\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.170072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.170042 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istiod-ca-cert\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.171374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.170291 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-data\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.171374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.170866 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-credential-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.171374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.170959 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.176951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.176926 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-envoy\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.195780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.189074 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-podinfo\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.198066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.197998 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-token\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.223353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.213221 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqwf\" (UniqueName: \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-kube-api-access-8rqwf\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.379488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.372816 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:48.430430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.430290 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:48.608784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.564960 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:48.792759 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:11:48.792724 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bcdc21_0f01_4986_a3d4_1a81611835f4.slice/crio-799c51382a3667e7d7847ba15277a91b3509c0f998a0a3112660a1ca955d8d01 WatchSource:0}: Error finding container 799c51382a3667e7d7847ba15277a91b3509c0f998a0a3112660a1ca955d8d01: Status 404 returned error can't find the container with id 799c51382a3667e7d7847ba15277a91b3509c0f998a0a3112660a1ca955d8d01 Oct 30 13:11:48.792929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.792904 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b"] Oct 30 13:11:48.796184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.796147 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:48.796304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.796224 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:48.796304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:48.796263 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:49.200451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:49.199103 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" event={"ID":"b4bcdc21-0f01-4986-a3d4-1a81611835f4","Type":"ContainerStarted","Data":"ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8"} Oct 30 13:11:49.200451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:49.199166 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" event={"ID":"b4bcdc21-0f01-4986-a3d4-1a81611835f4","Type":"ContainerStarted","Data":"799c51382a3667e7d7847ba15277a91b3509c0f998a0a3112660a1ca955d8d01"} Oct 30 13:11:49.336340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:49.335699 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" podStartSLOduration=2.33568073 podStartE2EDuration="2.33568073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:49.309741784 +0000 UTC m=+1216.124936233" watchObservedRunningTime="2026-10-30 13:11:49.33568073 +0000 UTC m=+1216.150875154" Oct 30 13:11:49.432489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:49.432449 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:49.434118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:49.434087 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:11:49.434264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:49.434139 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 13:11:50.502346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:50.501211 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:50.602485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:50.602448 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:51.448081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:51.448046 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:51.448439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:51.448408 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:11:51.934345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:51.931861 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:52.098812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:52.098776 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:52.181476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:52.181428 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:52.251376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:52.251244 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:52.256352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:52.253565 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:52.256352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:52.254407 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:11:52.424599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:52.424554 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:52.504425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:52.503423 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:52.516353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:52.513400 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:52.625352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:52.622419 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:11:52.839357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:52.838545 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:53.189516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.189410 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:11:53.190710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.190686 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:11:53.256366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.253591 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:53.256366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.254139 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:11:53.256366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.255911 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:53.256366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.256194 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 13:11:53.256728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.256694 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:11:53.256793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.256745 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:11:53.449618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.433788 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:53.456384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.450424 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:53.507453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.506430 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:53.686346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.623489 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:53.735491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.688056 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:11:53.735491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.688095 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 30 13:11:53.783344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.772360 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:11:53.783344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.772448 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:11:53.783344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.772489 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:11:53.783344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.772531 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:11:53.841378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.841333 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:53.844070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:53.844044 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:11:54.021392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:54.016055 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:54.232347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:54.192673 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:54.304350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:54.302450 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:54.340347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:54.322338 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:54.409348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:54.384082 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:11:54.703400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:54.481688 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:54.703400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:54.662427 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:54.871383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:54.871343 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:11:54.872081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:54.872046 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:54.901457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:54.880554 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:55.101133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.101086 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded" Oct 30 13:11:55.233679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.233636 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:11:55.318351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.316668 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:55.337376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.334410 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:55.683897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.683851 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:11:55.699511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.699472 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:55.699829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.699787 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:55.700274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.699729 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:55.877583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.877514 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:11:55.878975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.878936 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:55.891781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:55.891724 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:11:56.222397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:56.215728 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.333365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:56.326602 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:56.665347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:56.600205 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 13:11:56.715584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:56.715513 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:11:56.761345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:56.736664 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.846351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:56.816217 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:11:56.874471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:56.826388 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:56.883351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:56.879393 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:11:56.928537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:56.928434 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:56.941095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:56.882390 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:57.109347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.091844 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": context deadline exceeded" Oct 30 13:11:57.162468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.162417 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:57.239645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.239511 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 13:11:57.256007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.255865 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh"] Oct 30 13:11:57.284370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.284179 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 13:11:57.383444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.361101 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:57.603359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.601401 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:57.665346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.645602 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:11:57.687348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.677207 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5a6281\"" Oct 30 13:11:57.701429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.685119 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5a6281-dockercfg-ncq24\"" Oct 30 13:11:57.867400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.752873 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:57.894748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.809732 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b1cb66-7012-4411-ad37-b8fe9e24b586-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh\" (UID: \"45b1cb66-7012-4411-ad37-b8fe9e24b586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:11:57.894748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.889238 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4kz4\" (UniqueName: \"kubernetes.io/projected/45b1cb66-7012-4411-ad37-b8fe9e24b586-kube-api-access-h4kz4\") pod \"payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh\" (UID: \"45b1cb66-7012-4411-ad37-b8fe9e24b586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:11:57.894748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.837667 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh"] Oct 30 13:11:57.894748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.892893 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n"] Oct 30 13:11:57.896383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.881057 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:11:57.896383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.873561 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:57.898274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.881195 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:57.912653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.912630 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:11:57.929438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:57.928930 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:58.066459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.066420 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b1cb66-7012-4411-ad37-b8fe9e24b586-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh\" (UID: \"45b1cb66-7012-4411-ad37-b8fe9e24b586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:11:58.092727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.088050 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b1cb66-7012-4411-ad37-b8fe9e24b586-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh\" (UID: \"45b1cb66-7012-4411-ad37-b8fe9e24b586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:11:58.093557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.093440 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4kz4\" (UniqueName: \"kubernetes.io/projected/45b1cb66-7012-4411-ad37-b8fe9e24b586-kube-api-access-h4kz4\") pod \"payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh\" (UID: \"45b1cb66-7012-4411-ad37-b8fe9e24b586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:11:58.093778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.093749 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-947db\" (UniqueName: \"kubernetes.io/projected/9a90fb13-284a-4c86-b7b9-2598d28a392e-kube-api-access-947db\") pod \"payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n\" (UID: \"9a90fb13-284a-4c86-b7b9-2598d28a392e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:11:58.095128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.094812 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n"] Oct 30 13:11:58.095600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.095401 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a90fb13-284a-4c86-b7b9-2598d28a392e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n\" (UID: \"9a90fb13-284a-4c86-b7b9-2598d28a392e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:11:58.224041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.197920 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4kz4\" (UniqueName: \"kubernetes.io/projected/45b1cb66-7012-4411-ad37-b8fe9e24b586-kube-api-access-h4kz4\") pod \"payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh\" (UID: \"45b1cb66-7012-4411-ad37-b8fe9e24b586\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:11:58.224041 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:11:58.198226 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:11:58.224041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.206673 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:11:58.230730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.230490 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:58.286536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.258987 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a90fb13-284a-4c86-b7b9-2598d28a392e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n\" (UID: \"9a90fb13-284a-4c86-b7b9-2598d28a392e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:11:58.287487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.287447 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-947db\" (UniqueName: \"kubernetes.io/projected/9a90fb13-284a-4c86-b7b9-2598d28a392e-kube-api-access-947db\") pod \"payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n\" (UID: \"9a90fb13-284a-4c86-b7b9-2598d28a392e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:11:58.288164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.260134 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:11:58.288572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.260182 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:11:58.288982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.261577 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:11:58.289803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.289784 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:11:58.291763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.259936 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a90fb13-284a-4c86-b7b9-2598d28a392e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n\" (UID: \"9a90fb13-284a-4c86-b7b9-2598d28a392e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:11:58.421407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.398543 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f"] Oct 30 13:11:58.518729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.451870 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.518729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.467863 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:11:58.585703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.585372 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f"] Oct 30 13:11:58.612345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.586186 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ct4x\" (UniqueName: \"kubernetes.io/projected/13774796-6d31-40f4-8f3f-f548acf419a3-kube-api-access-6ct4x\") pod \"maas-api-revoke-keys-e2e-derive-45c4d25f-v876f\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.612345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.586277 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-45c4d25f-v876f\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.612345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.586394 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-45c4d25f-v876f\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.627502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.627445 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:58.640957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.640872 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:11:58.730854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.730817 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-45c4d25f-v876f\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.731146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.731120 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-45c4d25f-v876f\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.731655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.731632 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ct4x\" (UniqueName: \"kubernetes.io/projected/13774796-6d31-40f4-8f3f-f548acf419a3-kube-api-access-6ct4x\") pod \"maas-api-revoke-keys-e2e-derive-45c4d25f-v876f\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.733095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.733053 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-45c4d25f-v876f\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.800496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.747652 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r"] Oct 30 13:11:58.800496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.789042 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-45c4d25f-v876f\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.800496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.800290 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-947db\" (UniqueName: \"kubernetes.io/projected/9a90fb13-284a-4c86-b7b9-2598d28a392e-kube-api-access-947db\") pod \"payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n\" (UID: \"9a90fb13-284a-4c86-b7b9-2598d28a392e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:11:58.810672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.810608 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ct4x\" (UniqueName: \"kubernetes.io/projected/13774796-6d31-40f4-8f3f-f548acf419a3-kube-api-access-6ct4x\") pod \"maas-api-revoke-keys-e2e-derive-45c4d25f-v876f\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.839645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.839613 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:11:58.839884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.839868 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r"] Oct 30 13:11:58.841030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.841006 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:11:58.896151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.896112 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:58.933870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.901658 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:11:58.933870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.904818 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:58.933870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.929892 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:11:58.933870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.932725 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:11:58.970345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.963826 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:11:59.002399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.972586 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26nfj\" (UniqueName: \"kubernetes.io/projected/12a1d150-83a2-45f2-87e8-b2239c9eff8e-kube-api-access-26nfj\") pod \"maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:11:59.002399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:58.998498 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:11:59.002399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.000582 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:11:59.097299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.097259 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:59.175042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.171755 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:11:59.188364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.188130 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:11:59.192355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.189716 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:11:59.224409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.207079 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:11:59.224409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.207134 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26nfj\" (UniqueName: \"kubernetes.io/projected/12a1d150-83a2-45f2-87e8-b2239c9eff8e-kube-api-access-26nfj\") pod \"maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:11:59.224409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.207248 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:11:59.224409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.214864 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 13:11:59.253821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.253790 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:11:59.286751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.286703 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:11:59.371838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.290925 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:11:59.454345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.433815 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:59.587281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.587246 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:11:59.641115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.638050 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26nfj\" (UniqueName: \"kubernetes.io/projected/12a1d150-83a2-45f2-87e8-b2239c9eff8e-kube-api-access-26nfj\") pod \"maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:11:59.673931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.673895 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:11:59.791403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:11:59.789900 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:12:00.290569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:00.290523 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh"] Oct 30 13:12:00.352105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:00.346627 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:00.565482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:00.565436 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n"] Oct 30 13:12:00.647738 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:12:00.644025 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a1d150_83a2_45f2_87e8_b2239c9eff8e.slice/crio-0ee732644f4c37224824d08e52b6a72540d519e979883743d83c89cdcbd44e29 WatchSource:0}: Error finding container 0ee732644f4c37224824d08e52b6a72540d519e979883743d83c89cdcbd44e29: Status 404 returned error can't find the container with id 0ee732644f4c37224824d08e52b6a72540d519e979883743d83c89cdcbd44e29 Oct 30 13:12:00.661513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:00.661282 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" event={"ID":"45b1cb66-7012-4411-ad37-b8fe9e24b586","Type":"ContainerStarted","Data":"e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e"} Oct 30 13:12:00.661513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:00.661351 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" event={"ID":"45b1cb66-7012-4411-ad37-b8fe9e24b586","Type":"ContainerStarted","Data":"6b5c7e40a3feb57678355dd5ef10550265093b41106c10b89b06fa74c24aae62"} Oct 30 13:12:00.662350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:00.662292 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:12:00.673366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:00.664546 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r"] Oct 30 13:12:00.673366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:00.671797 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" event={"ID":"9a90fb13-284a-4c86-b7b9-2598d28a392e","Type":"ContainerStarted","Data":"a73db91e415fa71ae4cfece9009991e39b05aa97d52f8ecb56b56824eb9b5438"} Oct 30 13:12:00.677808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:00.677783 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f"] Oct 30 13:12:00.704888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:00.704846 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" podStartSLOduration=4.704833734 podStartE2EDuration="4.704833734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:00.701513337 +0000 UTC m=+1227.516707764" watchObservedRunningTime="2026-10-30 13:12:00.704833734 +0000 UTC m=+1227.520028159" Oct 30 13:12:01.683515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:01.683480 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" event={"ID":"9a90fb13-284a-4c86-b7b9-2598d28a392e","Type":"ContainerStarted","Data":"bf117999f27b80603f58c7e43cfbde0baf4b878b621a7bc2e64dbef8394a1285"} Oct 30 13:12:01.683968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:01.683550 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:12:01.685388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:01.685360 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" event={"ID":"13774796-6d31-40f4-8f3f-f548acf419a3","Type":"ContainerStarted","Data":"8059f92530acd112a570b38fcfcd31b7ceea764960b2b931e2e46d124f46a13a"} Oct 30 13:12:01.685499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:01.685410 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" event={"ID":"13774796-6d31-40f4-8f3f-f548acf419a3","Type":"ContainerStarted","Data":"2f30bfdce5954f995ba0ded6980c660a95274824037b39b59595ffa53d9bf334"} Oct 30 13:12:01.688450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:01.688427 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" event={"ID":"12a1d150-83a2-45f2-87e8-b2239c9eff8e","Type":"ContainerStarted","Data":"3e39340158cc6931739522810a28b7bbbdd9524ab4a63f9381274ca9b8088958"} Oct 30 13:12:01.688576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:01.688471 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" event={"ID":"12a1d150-83a2-45f2-87e8-b2239c9eff8e","Type":"ContainerStarted","Data":"0ee732644f4c37224824d08e52b6a72540d519e979883743d83c89cdcbd44e29"} Oct 30 13:12:01.734342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:01.733512 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" podStartSLOduration=5.733493195 podStartE2EDuration="5.733493195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:01.72397732 +0000 UTC m=+1228.539171748" watchObservedRunningTime="2026-10-30 13:12:01.733493195 +0000 UTC m=+1228.548687620" Oct 30 13:12:01.767184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:01.767123 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" podStartSLOduration=4.767108996 podStartE2EDuration="4.767108996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:01.761044374 +0000 UTC m=+1228.576238804" watchObservedRunningTime="2026-10-30 13:12:01.767108996 +0000 UTC m=+1228.582303419" Oct 30 13:12:01.805372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:01.805291 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" podStartSLOduration=5.805271301 podStartE2EDuration="5.805271301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:01.798019342 +0000 UTC m=+1228.613213765" watchObservedRunningTime="2026-10-30 13:12:01.805271301 +0000 UTC m=+1228.620465725" Oct 30 13:12:02.011895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:02.010422 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:02.701477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:02.701335 5049 generic.go:356] "Generic (PLEG): container finished" podID="13774796-6d31-40f4-8f3f-f548acf419a3" containerID="8059f92530acd112a570b38fcfcd31b7ceea764960b2b931e2e46d124f46a13a" exitCode=0 Oct 30 13:12:02.702170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:02.702060 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" event={"ID":"13774796-6d31-40f4-8f3f-f548acf419a3","Type":"ContainerDied","Data":"8059f92530acd112a570b38fcfcd31b7ceea764960b2b931e2e46d124f46a13a"} Oct 30 13:12:02.706624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:02.706529 5049 generic.go:356] "Generic (PLEG): container finished" podID="12a1d150-83a2-45f2-87e8-b2239c9eff8e" containerID="3e39340158cc6931739522810a28b7bbbdd9524ab4a63f9381274ca9b8088958" exitCode=0 Oct 30 13:12:02.706865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:02.706793 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" event={"ID":"12a1d150-83a2-45f2-87e8-b2239c9eff8e","Type":"ContainerDied","Data":"3e39340158cc6931739522810a28b7bbbdd9524ab4a63f9381274ca9b8088958"} Oct 30 13:12:03.904962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:03.904099 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:04.112846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.112809 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:12:04.160689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.160651 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:12:04.241396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.241020 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-openshift-ca\") pod \"13774796-6d31-40f4-8f3f-f548acf419a3\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " Oct 30 13:12:04.241396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.241100 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-openshift-ca\") pod \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " Oct 30 13:12:04.241396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.241138 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13774796-6d31-40f4-8f3f-f548acf419a3-kube-api-access-6ct4x\" (UniqueName: \"kubernetes.io/projected/13774796-6d31-40f4-8f3f-f548acf419a3-kube-api-access-6ct4x\") pod \"13774796-6d31-40f4-8f3f-f548acf419a3\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " Oct 30 13:12:04.241396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.241164 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a1d150-83a2-45f2-87e8-b2239c9eff8e-kube-api-access-26nfj\" (UniqueName: \"kubernetes.io/projected/12a1d150-83a2-45f2-87e8-b2239c9eff8e-kube-api-access-26nfj\") pod \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " Oct 30 13:12:04.241396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.241363 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-xks-ca\") pod \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\" (UID: \"12a1d150-83a2-45f2-87e8-b2239c9eff8e\") " Oct 30 13:12:04.241793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.241415 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-xks-ca\") pod \"13774796-6d31-40f4-8f3f-f548acf419a3\" (UID: \"13774796-6d31-40f4-8f3f-f548acf419a3\") " Oct 30 13:12:04.261423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.254015 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-openshift-ca" pod "13774796-6d31-40f4-8f3f-f548acf419a3" (UID: "13774796-6d31-40f4-8f3f-f548acf419a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:04.261423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.254356 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-openshift-ca" pod "12a1d150-83a2-45f2-87e8-b2239c9eff8e" (UID: "12a1d150-83a2-45f2-87e8-b2239c9eff8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:04.261423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.260939 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a1d150-83a2-45f2-87e8-b2239c9eff8e-kube-api-access-26nfj" pod "12a1d150-83a2-45f2-87e8-b2239c9eff8e" (UID: "12a1d150-83a2-45f2-87e8-b2239c9eff8e"). InnerVolumeSpecName "kube-api-access-26nfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:04.262129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.262104 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13774796-6d31-40f4-8f3f-f548acf419a3-kube-api-access-6ct4x" pod "13774796-6d31-40f4-8f3f-f548acf419a3" (UID: "13774796-6d31-40f4-8f3f-f548acf419a3"). InnerVolumeSpecName "kube-api-access-6ct4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:04.262295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.262262 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-xks-ca" pod "12a1d150-83a2-45f2-87e8-b2239c9eff8e" (UID: "12a1d150-83a2-45f2-87e8-b2239c9eff8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:04.268395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.265419 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-xks-ca" pod "13774796-6d31-40f4-8f3f-f548acf419a3" (UID: "13774796-6d31-40f4-8f3f-f548acf419a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:04.344343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.343554 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:04.344343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.343591 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:04.344343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.343608 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ct4x\" (UniqueName: \"kubernetes.io/projected/13774796-6d31-40f4-8f3f-f548acf419a3-kube-api-access-6ct4x\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:04.344343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.343623 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26nfj\" (UniqueName: \"kubernetes.io/projected/12a1d150-83a2-45f2-87e8-b2239c9eff8e-kube-api-access-26nfj\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:04.344343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.343638 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a1d150-83a2-45f2-87e8-b2239c9eff8e-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:04.344343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.343653 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13774796-6d31-40f4-8f3f-f548acf419a3-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:04.735137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.735102 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" Oct 30 13:12:04.735137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.735119 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f" event={"ID":"13774796-6d31-40f4-8f3f-f548acf419a3","Type":"ContainerDied","Data":"2f30bfdce5954f995ba0ded6980c660a95274824037b39b59595ffa53d9bf334"} Oct 30 13:12:04.735433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.735186 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f30bfdce5954f995ba0ded6980c660a95274824037b39b59595ffa53d9bf334" Oct 30 13:12:04.739367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.737144 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" Oct 30 13:12:04.739367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.737179 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r" event={"ID":"12a1d150-83a2-45f2-87e8-b2239c9eff8e","Type":"ContainerDied","Data":"0ee732644f4c37224824d08e52b6a72540d519e979883743d83c89cdcbd44e29"} Oct 30 13:12:04.739367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:04.737241 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee732644f4c37224824d08e52b6a72540d519e979883743d83c89cdcbd44e29" Oct 30 13:12:05.630227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:05.629479 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:06.429666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.017015 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:06.762783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.762696 5049 generic.go:356] "Generic (PLEG): container finished" podID="89b711a4-1edf-4405-b7cc-b493017f2949" containerID="56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff" exitCode=0 Oct 30 13:12:06.762783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.762772 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" event={"ID":"89b711a4-1edf-4405-b7cc-b493017f2949","Type":"ContainerDied","Data":"56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff"} Oct 30 13:12:06.763331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.763262 5049 scope.go:122] "RemoveContainer" containerID="56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff" Oct 30 13:12:06.764730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.764707 5049 generic.go:356] "Generic (PLEG): container finished" podID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerID="66893a8d4e9570f84176c8883c8d1749c54a0c75bd02c94f6496e50bc8b92523" exitCode=0 Oct 30 13:12:06.764831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.764787 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" event={"ID":"d5a94ef0-413f-48c1-aeee-2538cb0afb2f","Type":"ContainerDied","Data":"66893a8d4e9570f84176c8883c8d1749c54a0c75bd02c94f6496e50bc8b92523"} Oct 30 13:12:06.765179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.765149 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:06.765254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.765237 5049 scope.go:122] "RemoveContainer" containerID="66893a8d4e9570f84176c8883c8d1749c54a0c75bd02c94f6496e50bc8b92523" Oct 30 13:12:06.765353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.765235 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:06.765498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.765391 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:06.767151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.767117 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:06.767243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.767202 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:06.767283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:06.767248 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:07.358867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:07.353793 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:07.521351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:07.517613 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:07.799521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:07.799491 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" event={"ID":"89b711a4-1edf-4405-b7cc-b493017f2949","Type":"ContainerStarted","Data":"89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5"} Oct 30 13:12:07.801828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:07.801802 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" event={"ID":"d5a94ef0-413f-48c1-aeee-2538cb0afb2f","Type":"ContainerStarted","Data":"d5c207c58ba310336260f77dc060ae2bf5f0cba08ecea266c3f74400ffadf54a"} Oct 30 13:12:07.832871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:07.832834 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:07.862204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:07.862121 5049 patch_prober.go:29] interesting pod/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:12:07.862204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:07.862191 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 13:12:07.862521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:07.862498 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:12:08.438357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.431918 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:08.612703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.596443 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:12:08.617394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.613548 5049 patch_prober.go:29] interesting pod/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:12:08.642138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.641796 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 13:12:08.731252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.731164 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:08.918581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.896585 5049 patch_prober.go:29] interesting pod/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:12:08.918581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.896642 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 13:12:08.918581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.902221 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:08.966520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.954793 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5"] Oct 30 13:12:08.966520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.956032 5049 cpu_manager.go:414] "containerMap: removing container" podUID="12a1d150-83a2-45f2-87e8-b2239c9eff8e" containerName="revoke-keys" Oct 30 13:12:08.966520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.956054 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a1d150-83a2-45f2-87e8-b2239c9eff8e" containerName="revoke-keys" Oct 30 13:12:08.966520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.956101 5049 cpu_manager.go:414] "containerMap: removing container" podUID="13774796-6d31-40f4-8f3f-f548acf419a3" containerName="revoke-keys" Oct 30 13:12:08.966520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.956109 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13774796-6d31-40f4-8f3f-f548acf419a3" containerName="revoke-keys" Oct 30 13:12:08.974367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.970555 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a1d150-83a2-45f2-87e8-b2239c9eff8e" containerName="revoke-keys" Oct 30 13:12:08.974367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:08.970601 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="13774796-6d31-40f4-8f3f-f548acf419a3" containerName="revoke-keys" Oct 30 13:12:09.005454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.001308 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.102238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.091015 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-df6c3661-openshift-default-dockercfg-hfkrp\"" Oct 30 13:12:09.103421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.103389 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-certs\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.119648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.119612 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrxn\" (UniqueName: \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-kube-api-access-hrrxn\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.120060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.120005 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-data\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.120344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.120307 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.120934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.120908 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-podinfo\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.144596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.144561 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-token\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.158615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.145053 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-credential-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.179243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.159100 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-envoy\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.296584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.296537 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istiod-ca-cert\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.297053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.215872 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:09.297293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.175474 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5"] Oct 30 13:12:09.418716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.417543 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istiod-ca-cert\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.418716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.417674 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-certs\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.418716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.417732 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrxn\" (UniqueName: \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-kube-api-access-hrrxn\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.418716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.417795 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-data\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.418716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.417851 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.418716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.417946 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-podinfo\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.418716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.417972 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-token\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.418716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.418051 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-credential-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.418716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.418155 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-envoy\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.447376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.421825 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-data\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.447376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.422486 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istiod-ca-cert\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.447376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.422741 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-certs\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.471410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.471374 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.506350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.484452 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrxn\" (UniqueName: \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-kube-api-access-hrrxn\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.540343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.528656 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-podinfo\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.540343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.529353 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-credential-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.540343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.530486 5049 patch_prober.go:29] interesting pod/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:12:09.540343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.530553 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 13:12:09.604539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.604410 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-envoy\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.608254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.607350 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:09.615368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.615282 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-token\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.639206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.638933 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:09.860102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.859796 5049 patch_prober.go:29] interesting pod/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:12:09.860102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.859854 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 13:12:09.866234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.866075 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:09.911919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:09.911874 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:12:10.460398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:10.458831 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5"] Oct 30 13:12:10.466942 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:12:10.466903 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb240b8_72ef_4031_aff9_c7ff4707a0aa.slice/crio-779609fef91f24597d97b509075f0798c9b03ef89a35b3f2c92461bb714f3528 WatchSource:0}: Error finding container 779609fef91f24597d97b509075f0798c9b03ef89a35b3f2c92461bb714f3528: Status 404 returned error can't find the container with id 779609fef91f24597d97b509075f0798c9b03ef89a35b3f2c92461bb714f3528 Oct 30 13:12:10.469700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:10.469667 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:10.469773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:10.469749 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:10.469827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:10.469780 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:10.532345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:10.524350 5049 patch_prober.go:29] interesting pod/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:12:10.532345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:10.524431 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 13:12:10.640471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:10.640428 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:12:10.666342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:10.642452 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:12:10.872367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:10.864706 5049 patch_prober.go:29] interesting pod/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:12:10.872367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:10.864782 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 13:12:11.077765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:11.077722 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" event={"ID":"dbb240b8-72ef-4031-aff9-c7ff4707a0aa","Type":"ContainerStarted","Data":"8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc"} Oct 30 13:12:11.077919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:11.077795 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" event={"ID":"dbb240b8-72ef-4031-aff9-c7ff4707a0aa","Type":"ContainerStarted","Data":"779609fef91f24597d97b509075f0798c9b03ef89a35b3f2c92461bb714f3528"} Oct 30 13:12:11.143360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:11.135144 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" podStartSLOduration=3.135122265 podStartE2EDuration="3.135122265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:11.121050325 +0000 UTC m=+1237.936244749" watchObservedRunningTime="2026-10-30 13:12:11.135122265 +0000 UTC m=+1237.950316688" Oct 30 13:12:11.240349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:11.237223 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:11.527400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:11.525686 5049 patch_prober.go:29] interesting pod/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:12:11.527400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:11.525747 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 13:12:11.659343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:11.643301 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:11.659343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:11.653675 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:11.865215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:11.863825 5049 patch_prober.go:29] interesting pod/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:12:11.865215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:11.863899 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 13:12:12.087382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:12.087354 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:12.088350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:12.088299 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:12:12.526345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:12.519705 5049 patch_prober.go:29] interesting pod/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:12:12.526345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:12.519782 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 13:12:12.667343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:12.661417 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:12.757345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:12.755511 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:12:12.776712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:12.776039 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:12:12.877448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:12.877415 5049 patch_prober.go:29] interesting pod/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:12:12.877804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:12.877755 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 13:12:12.995103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:12.993562 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp"] Oct 30 13:12:13.002462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.001732 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.002462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.002306 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp"] Oct 30 13:12:13.046400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.038868 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-5a6281-serving-cert\"" Oct 30 13:12:13.102017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.101911 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:13.104050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.104012 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5gr8\" (UniqueName: \"kubernetes.io/projected/5954f9fb-eadc-458f-a39d-678fc56e6e5e-kube-api-access-j5gr8\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.104203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.104157 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.104273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.104232 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5954f9fb-eadc-458f-a39d-678fc56e6e5e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.104462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.104442 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.395446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.395346 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5gr8\" (UniqueName: \"kubernetes.io/projected/5954f9fb-eadc-458f-a39d-678fc56e6e5e-kube-api-access-j5gr8\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.395624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.395525 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.395624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.395609 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5954f9fb-eadc-458f-a39d-678fc56e6e5e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.396494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.396452 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5954f9fb-eadc-458f-a39d-678fc56e6e5e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.397174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.396712 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.465383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.405643 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.467991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.467951 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:13.468753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.468727 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.470360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.470116 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5gr8\" (UniqueName: \"kubernetes.io/projected/5954f9fb-eadc-458f-a39d-678fc56e6e5e-kube-api-access-j5gr8\") pod \"maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.528769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.528726 5049 patch_prober.go:29] interesting pod/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:12:13.529448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.529414 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 13:12:13.679381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.679223 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:13.681822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.681793 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:13.876734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.876270 5049 patch_prober.go:29] interesting pod/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:12:13.876734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.876400 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 13:12:13.877832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:13.877774 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:14.067919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:14.067878 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:14.168760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:14.168731 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp"] Oct 30 13:12:14.400287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:14.400204 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" event={"ID":"5954f9fb-eadc-458f-a39d-678fc56e6e5e","Type":"ContainerStarted","Data":"ab09195ef53f3fb2ff1ad2c83e4ea50f37cd66afcd4887c41000542605cad62c"} Oct 30 13:12:14.537256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:14.537221 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:14.902344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:14.895558 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:15.327347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.302100 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:15.391384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.390195 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5"] Oct 30 13:12:15.391384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.390349 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:12:15.394349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.391990 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" podUID="471b846a-a62e-49fc-bb51-248d92dcc932" containerName="maas-api" containerID="cri-o://e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d" gracePeriod=30 Oct 30 13:12:15.394590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.394470 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:12:15.611577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.610418 5049 patch_prober.go:29] interesting pod/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:12:15.611577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.610495 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:15.613730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.613637 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" event={"ID":"5954f9fb-eadc-458f-a39d-678fc56e6e5e","Type":"ContainerStarted","Data":"5fdf11629156c8e8f3b1b5019663fae72cce27814ed037ad870ac3076e990b5d"} Oct 30 13:12:15.613933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.613892 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:15.642272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.633670 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:12:15.684292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.684262 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:12:15.751343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.724832 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:15.872674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.872547 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" podStartSLOduration=3.24006905 podStartE2EDuration="3.872524983s" totalImagesPullingTime="632.455933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:15.829920344 +0000 UTC m=+1242.645114771" watchObservedRunningTime="2026-10-30 13:12:15.872524983 +0000 UTC m=+1242.687719407" Oct 30 13:12:15.977342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:15.977073 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:16.068689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.068657 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r"] Oct 30 13:12:16.076410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.076375 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-911fe1d6-b8n2r"] Oct 30 13:12:16.094463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.094430 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:12:16.146765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.146566 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:16.172966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.172931 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/471b846a-a62e-49fc-bb51-248d92dcc932-kube-api-access-rm2sc\" (UniqueName: \"kubernetes.io/projected/471b846a-a62e-49fc-bb51-248d92dcc932-kube-api-access-rm2sc\") pod \"471b846a-a62e-49fc-bb51-248d92dcc932\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " Oct 30 13:12:16.173186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.173012 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-maas-api-tls\") pod \"471b846a-a62e-49fc-bb51-248d92dcc932\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " Oct 30 13:12:16.173186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.173044 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/471b846a-a62e-49fc-bb51-248d92dcc932-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/471b846a-a62e-49fc-bb51-248d92dcc932-openshift-service-ca\") pod \"471b846a-a62e-49fc-bb51-248d92dcc932\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " Oct 30 13:12:16.173186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.173166 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-metrics-certs\" (UniqueName: \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-metrics-certs\") pod \"471b846a-a62e-49fc-bb51-248d92dcc932\" (UID: \"471b846a-a62e-49fc-bb51-248d92dcc932\") " Oct 30 13:12:16.173910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.173845 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471b846a-a62e-49fc-bb51-248d92dcc932-openshift-service-ca" pod "471b846a-a62e-49fc-bb51-248d92dcc932" (UID: "471b846a-a62e-49fc-bb51-248d92dcc932"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:16.174113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.174089 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/471b846a-a62e-49fc-bb51-248d92dcc932-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:16.175525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.175498 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471b846a-a62e-49fc-bb51-248d92dcc932-kube-api-access-rm2sc" pod "471b846a-a62e-49fc-bb51-248d92dcc932" (UID: "471b846a-a62e-49fc-bb51-248d92dcc932"). InnerVolumeSpecName "kube-api-access-rm2sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:16.175757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.175733 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-maas-api-tls" pod "471b846a-a62e-49fc-bb51-248d92dcc932" (UID: "471b846a-a62e-49fc-bb51-248d92dcc932"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:16.176004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.175979 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-metrics-certs" pod "471b846a-a62e-49fc-bb51-248d92dcc932" (UID: "471b846a-a62e-49fc-bb51-248d92dcc932"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:16.416485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.408818 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm2sc\" (UniqueName: \"kubernetes.io/projected/471b846a-a62e-49fc-bb51-248d92dcc932-kube-api-access-rm2sc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:16.416729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.416702 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:16.416834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.416823 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471b846a-a62e-49fc-bb51-248d92dcc932-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:16.482342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.465204 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:16.637217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.637182 5049 generic.go:356] "Generic (PLEG): container finished" podID="471b846a-a62e-49fc-bb51-248d92dcc932" containerID="e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d" exitCode=0 Oct 30 13:12:16.638151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.638132 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" Oct 30 13:12:16.642565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.642525 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" event={"ID":"471b846a-a62e-49fc-bb51-248d92dcc932","Type":"ContainerDied","Data":"e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d"} Oct 30 13:12:16.642787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.642757 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5" event={"ID":"471b846a-a62e-49fc-bb51-248d92dcc932","Type":"ContainerDied","Data":"784b62d82c535a8fbd6e488bfa9227d8c9a3f8d67db32c67a0f6e8b1f0438ef5"} Oct 30 13:12:16.642937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.642921 5049 scope.go:122] "RemoveContainer" containerID="e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d" Oct 30 13:12:16.647174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.647108 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:16.681924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.681897 5049 scope.go:122] "RemoveContainer" containerID="e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d" Oct 30 13:12:16.682923 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:12:16.682892 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d\": container with ID starting with e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d not found: ID does not exist" containerID="e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d" Oct 30 13:12:16.683124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.683099 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d"} err="failed to get container status \"e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d\": rpc error: code = NotFound desc = could not find container \"e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d\": container with ID starting with e76f2c83b0705aa4a43e2a2c3ca9439630cf2cf9184bb9823bb450d854143f3d not found: ID does not exist" Oct 30 13:12:16.685347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.685306 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f"] Oct 30 13:12:16.699893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.699858 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45c4d25f-v876f"] Oct 30 13:12:16.719897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.719866 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5"] Oct 30 13:12:16.735303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.735268 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-911fe1d6-7d9788c49-8snn5"] Oct 30 13:12:16.815431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.815373 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9"] Oct 30 13:12:16.902053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:16.902008 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:17.347055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:17.346915 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a1d150-83a2-45f2-87e8-b2239c9eff8e" path="/var/lib/kubelet/pods/12a1d150-83a2-45f2-87e8-b2239c9eff8e/volumes" Oct 30 13:12:17.348527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:17.348500 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13774796-6d31-40f4-8f3f-f548acf419a3" path="/var/lib/kubelet/pods/13774796-6d31-40f4-8f3f-f548acf419a3/volumes" Oct 30 13:12:17.349135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:17.349111 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="471b846a-a62e-49fc-bb51-248d92dcc932" path="/var/lib/kubelet/pods/471b846a-a62e-49fc-bb51-248d92dcc932/volumes" Oct 30 13:12:17.683384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:17.683251 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" containerID="cri-o://89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5" gracePeriod=30 Oct 30 13:12:17.706888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:17.702245 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl"] Oct 30 13:12:17.706888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:17.702563 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" containerID="cri-o://d5c207c58ba310336260f77dc060ae2bf5f0cba08ecea266c3f74400ffadf54a" gracePeriod=30 Oct 30 13:12:17.758849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:17.758816 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:18.049430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:18.043406 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6"] Oct 30 13:12:18.049430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:18.048993 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1ab4ee-dn9m6"] Oct 30 13:12:18.515904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:18.515818 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:18.680346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:18.679713 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:18.873274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:18.872937 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:19.097244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.097203 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf"] Oct 30 13:12:19.098432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.098409 5049 cpu_manager.go:414] "containerMap: removing container" podUID="471b846a-a62e-49fc-bb51-248d92dcc932" containerName="maas-api" Oct 30 13:12:19.098432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.098430 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="471b846a-a62e-49fc-bb51-248d92dcc932" containerName="maas-api" Oct 30 13:12:19.098662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.098648 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="471b846a-a62e-49fc-bb51-248d92dcc932" containerName="maas-api" Oct 30 13:12:19.105176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.105156 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.135247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.135192 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-f0a31b-openshift-default-dockercfg-txvm6\"" Oct 30 13:12:19.146269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.146236 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf"] Oct 30 13:12:19.158684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.158651 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:19.174470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.174437 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gkm7\" (UniqueName: \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-kube-api-access-2gkm7\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.174756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.174735 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-envoy\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.174997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.174979 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-credential-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.175179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.175164 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-data\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.175420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.175402 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.175580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.175566 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e3f959-a61b-4e80-9437-4597939fe709-istiod-ca-cert\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.175699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.175685 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-certs\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.175804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.175790 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e3f959-a61b-4e80-9437-4597939fe709-istio-podinfo\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.175962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.175947 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-istio-token\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.297346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.284772 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-data\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.309128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.298995 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-data\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.313959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.313893 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.314154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.309055 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.314373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.314354 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e3f959-a61b-4e80-9437-4597939fe709-istiod-ca-cert\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.314535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.314518 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-certs\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.314660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.314646 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e3f959-a61b-4e80-9437-4597939fe709-istio-podinfo\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.314864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.314850 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-istio-token\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.315135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.315119 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gkm7\" (UniqueName: \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-kube-api-access-2gkm7\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.317051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.315918 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e3f959-a61b-4e80-9437-4597939fe709-istiod-ca-cert\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.317407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.317381 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-envoy\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.318025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.318006 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-certs\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.318442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.318425 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-credential-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.319112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.319092 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-credential-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.380583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.380536 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:19.417790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.417692 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485de4c0-2624-4e32-b606-18088c985410" path="/var/lib/kubelet/pods/485de4c0-2624-4e32-b606-18088c985410/volumes" Oct 30 13:12:19.419246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.419220 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e3f959-a61b-4e80-9437-4597939fe709-istio-podinfo\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.430076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.430047 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gkm7\" (UniqueName: \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-kube-api-access-2gkm7\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.435133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.435071 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-istio-token\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.443400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.443334 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-envoy\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.451947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.451666 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:19.677653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.677491 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:19.889651 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:12:19.889622 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e3f959_a61b_4e80_9437_4597939fe709.slice/crio-5998e384f6453244f9b8a4334188e5dab85018c12d3654b47dc50426186228fd WatchSource:0}: Error finding container 5998e384f6453244f9b8a4334188e5dab85018c12d3654b47dc50426186228fd: Status 404 returned error can't find the container with id 5998e384f6453244f9b8a4334188e5dab85018c12d3654b47dc50426186228fd Oct 30 13:12:19.891941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.891917 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf"] Oct 30 13:12:19.892851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.892816 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:19.892956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.892907 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:19.893022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:19.892952 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:20.392954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.392917 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:20.525958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.525914 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7"] Oct 30 13:12:20.531008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.530982 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:12:20.563032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.562993 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7"] Oct 30 13:12:20.633391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.614846 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3778dacd-dockercfg-27f7x\"" Oct 30 13:12:20.633391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.630422 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:20.656412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.635466 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3778dacd\"" Oct 30 13:12:20.729363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.729301 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925e38f7-fc9d-4d6c-9c00-660ebca23641-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7\" (UID: \"925e38f7-fc9d-4d6c-9c00-660ebca23641\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:12:20.729870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.729397 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66xr\" (UniqueName: \"kubernetes.io/projected/925e38f7-fc9d-4d6c-9c00-660ebca23641-kube-api-access-g66xr\") pod \"payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7\" (UID: \"925e38f7-fc9d-4d6c-9c00-660ebca23641\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:12:20.833538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.833488 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925e38f7-fc9d-4d6c-9c00-660ebca23641-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7\" (UID: \"925e38f7-fc9d-4d6c-9c00-660ebca23641\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:12:20.834932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.833823 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66xr\" (UniqueName: \"kubernetes.io/projected/925e38f7-fc9d-4d6c-9c00-660ebca23641-kube-api-access-g66xr\") pod \"payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7\" (UID: \"925e38f7-fc9d-4d6c-9c00-660ebca23641\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:12:20.836421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.836394 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925e38f7-fc9d-4d6c-9c00-660ebca23641-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7\" (UID: \"925e38f7-fc9d-4d6c-9c00-660ebca23641\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:12:20.957049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:20.902906 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66xr\" (UniqueName: \"kubernetes.io/projected/925e38f7-fc9d-4d6c-9c00-660ebca23641-kube-api-access-g66xr\") pod \"payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7\" (UID: \"925e38f7-fc9d-4d6c-9c00-660ebca23641\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:12:21.022256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:21.022198 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" event={"ID":"a2e3f959-a61b-4e80-9437-4597939fe709","Type":"ContainerStarted","Data":"27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830"} Oct 30 13:12:21.022448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:21.022283 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" event={"ID":"a2e3f959-a61b-4e80-9437-4597939fe709","Type":"ContainerStarted","Data":"5998e384f6453244f9b8a4334188e5dab85018c12d3654b47dc50426186228fd"} Oct 30 13:12:21.066271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:21.066032 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" podStartSLOduration=2.06601525 podStartE2EDuration="2.06601525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:21.0656166 +0000 UTC m=+1247.880811023" watchObservedRunningTime="2026-10-30 13:12:21.06601525 +0000 UTC m=+1247.881209673" Oct 30 13:12:21.152336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:21.152286 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:12:21.452378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:21.452343 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:21.458039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:21.458013 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:21.661723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:21.659935 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:21.674008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:21.673968 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7"] Oct 30 13:12:21.685843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:21.685803 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:12:22.035391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:22.035277 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" event={"ID":"925e38f7-fc9d-4d6c-9c00-660ebca23641","Type":"ContainerStarted","Data":"0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d"} Oct 30 13:12:22.035391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:22.035364 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:12:22.035391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:22.035391 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" event={"ID":"925e38f7-fc9d-4d6c-9c00-660ebca23641","Type":"ContainerStarted","Data":"06b234696328dabd5e2572d14952f67c3d26320dee57e08ed1ca5fb8037a14ae"} Oct 30 13:12:22.035938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:22.035793 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:22.037012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:22.036991 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:12:22.079233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:22.079184 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" podStartSLOduration=2.079170869 podStartE2EDuration="2.079170869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:22.071905781 +0000 UTC m=+1248.887100204" watchObservedRunningTime="2026-10-30 13:12:22.079170869 +0000 UTC m=+1248.894365329" Oct 30 13:12:22.937882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:22.937854 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:12:23.034885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.034402 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-istio-token\" (UniqueName: \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-istio-token\") pod \"89b711a4-1edf-4405-b7cc-b493017f2949\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " Oct 30 13:12:23.034885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.034477 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-certs\") pod \"89b711a4-1edf-4405-b7cc-b493017f2949\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " Oct 30 13:12:23.034885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.034620 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-kube-api-access-j2wh6\" (UniqueName: \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-kube-api-access-j2wh6\") pod \"89b711a4-1edf-4405-b7cc-b493017f2949\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " Oct 30 13:12:23.034885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.034655 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89b711a4-1edf-4405-b7cc-b493017f2949-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b711a4-1edf-4405-b7cc-b493017f2949-istio-podinfo\") pod \"89b711a4-1edf-4405-b7cc-b493017f2949\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " Oct 30 13:12:23.034885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.034793 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-data\") pod \"89b711a4-1edf-4405-b7cc-b493017f2949\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " Oct 30 13:12:23.034885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.034866 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-socket\") pod \"89b711a4-1edf-4405-b7cc-b493017f2949\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " Oct 30 13:12:23.035360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.034909 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-credential-socket\") pod \"89b711a4-1edf-4405-b7cc-b493017f2949\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " Oct 30 13:12:23.035360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.035035 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b711a4-1edf-4405-b7cc-b493017f2949-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b711a4-1edf-4405-b7cc-b493017f2949-istiod-ca-cert\") pod \"89b711a4-1edf-4405-b7cc-b493017f2949\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " Oct 30 13:12:23.035360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.035079 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-envoy\") pod \"89b711a4-1edf-4405-b7cc-b493017f2949\" (UID: \"89b711a4-1edf-4405-b7cc-b493017f2949\") " Oct 30 13:12:23.035947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.035559 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-socket" pod "89b711a4-1edf-4405-b7cc-b493017f2949" (UID: "89b711a4-1edf-4405-b7cc-b493017f2949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:23.036004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.035945 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.036072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.036047 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-certs" pod "89b711a4-1edf-4405-b7cc-b493017f2949" (UID: "89b711a4-1edf-4405-b7cc-b493017f2949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:23.036200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.036178 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-data" pod "89b711a4-1edf-4405-b7cc-b493017f2949" (UID: "89b711a4-1edf-4405-b7cc-b493017f2949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:23.039170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.036397 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-credential-socket" pod "89b711a4-1edf-4405-b7cc-b493017f2949" (UID: "89b711a4-1edf-4405-b7cc-b493017f2949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:23.039170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.036685 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b711a4-1edf-4405-b7cc-b493017f2949-istiod-ca-cert" pod "89b711a4-1edf-4405-b7cc-b493017f2949" (UID: "89b711a4-1edf-4405-b7cc-b493017f2949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:23.039170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.039137 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-envoy" pod "89b711a4-1edf-4405-b7cc-b493017f2949" (UID: "89b711a4-1edf-4405-b7cc-b493017f2949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:23.039498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.039182 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-istio-token" pod "89b711a4-1edf-4405-b7cc-b493017f2949" (UID: "89b711a4-1edf-4405-b7cc-b493017f2949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:23.041503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.041477 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89b711a4-1edf-4405-b7cc-b493017f2949-istio-podinfo" pod "89b711a4-1edf-4405-b7cc-b493017f2949" (UID: "89b711a4-1edf-4405-b7cc-b493017f2949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:23.042049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.042023 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-kube-api-access-j2wh6" pod "89b711a4-1edf-4405-b7cc-b493017f2949" (UID: "89b711a4-1edf-4405-b7cc-b493017f2949"). InnerVolumeSpecName "kube-api-access-j2wh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:23.051411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.051376 5049 generic.go:356] "Generic (PLEG): container finished" podID="89b711a4-1edf-4405-b7cc-b493017f2949" containerID="89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5" exitCode=0 Oct 30 13:12:23.051521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.051438 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" Oct 30 13:12:23.051521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.051476 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" event={"ID":"89b711a4-1edf-4405-b7cc-b493017f2949","Type":"ContainerDied","Data":"89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5"} Oct 30 13:12:23.051663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.051562 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9" event={"ID":"89b711a4-1edf-4405-b7cc-b493017f2949","Type":"ContainerDied","Data":"dcf0147c92d34e9b7855172855f5439eb6aa5efe6d40981eee057b7075d09cc9"} Oct 30 13:12:23.051663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.051611 5049 scope.go:122] "RemoveContainer" containerID="89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5" Oct 30 13:12:23.054179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.054152 5049 generic.go:356] "Generic (PLEG): container finished" podID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerID="d5c207c58ba310336260f77dc060ae2bf5f0cba08ecea266c3f74400ffadf54a" exitCode=0 Oct 30 13:12:23.054454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.054431 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" event={"ID":"d5a94ef0-413f-48c1-aeee-2538cb0afb2f","Type":"ContainerDied","Data":"d5c207c58ba310336260f77dc060ae2bf5f0cba08ecea266c3f74400ffadf54a"} Oct 30 13:12:23.078007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.077981 5049 scope.go:122] "RemoveContainer" containerID="56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff" Oct 30 13:12:23.080824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.080797 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:12:23.093270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.093249 5049 scope.go:122] "RemoveContainer" containerID="89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5" Oct 30 13:12:23.093630 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:12:23.093609 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5\": container with ID starting with 89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5 not found: ID does not exist" containerID="89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5" Oct 30 13:12:23.093714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.093638 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5"} err="failed to get container status \"89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5\": rpc error: code = NotFound desc = could not find container \"89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5\": container with ID starting with 89850cb4f78baa9b6737faa412c6159ffd6d8bcac681c9b8adf54e8f94a2daa5 not found: ID does not exist" Oct 30 13:12:23.093714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.093657 5049 scope.go:122] "RemoveContainer" containerID="56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff" Oct 30 13:12:23.093928 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:12:23.093912 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff\": container with ID starting with 56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff not found: ID does not exist" containerID="56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff" Oct 30 13:12:23.093967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.093934 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff"} err="failed to get container status \"56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff\": rpc error: code = NotFound desc = could not find container \"56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff\": container with ID starting with 56c7ff484cf44a59318c65d3c84d984a39b5c2841ae13ee72a71ace3452bd8ff not found: ID does not exist" Oct 30 13:12:23.093967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.093950 5049 scope.go:122] "RemoveContainer" containerID="66893a8d4e9570f84176c8883c8d1749c54a0c75bd02c94f6496e50bc8b92523" Oct 30 13:12:23.113865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.113838 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9"] Oct 30 13:12:23.128350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.125771 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-45c4d25f-openshift-default-7b698fd6f-2n7d9"] Oct 30 13:12:23.136665 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.136640 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-certs\") pod \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " Oct 30 13:12:23.136807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.136755 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istiod-ca-cert\") pod \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " Oct 30 13:12:23.136807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.136798 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-socket\") pod \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " Oct 30 13:12:23.136920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.136826 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-data\") pod \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " Oct 30 13:12:23.136976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.136916 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-podinfo\") pod \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " Oct 30 13:12:23.136976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.136935 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-certs" pod "d5a94ef0-413f-48c1-aeee-2538cb0afb2f" (UID: "d5a94ef0-413f-48c1-aeee-2538cb0afb2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:23.137087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.136997 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-envoy\") pod \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " Oct 30 13:12:23.137087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.137071 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-credential-socket\") pod \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " Oct 30 13:12:23.137191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.137146 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-socket" pod "d5a94ef0-413f-48c1-aeee-2538cb0afb2f" (UID: "d5a94ef0-413f-48c1-aeee-2538cb0afb2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:23.137191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.137159 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-data" pod "d5a94ef0-413f-48c1-aeee-2538cb0afb2f" (UID: "d5a94ef0-413f-48c1-aeee-2538cb0afb2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:23.137289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.137203 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istiod-ca-cert" pod "d5a94ef0-413f-48c1-aeee-2538cb0afb2f" (UID: "d5a94ef0-413f-48c1-aeee-2538cb0afb2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:23.137289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.137217 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-token\" (UniqueName: \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-token\") pod \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " Oct 30 13:12:23.137289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.137268 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-kube-api-access-plf8l\" (UniqueName: \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-kube-api-access-plf8l\") pod \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\" (UID: \"d5a94ef0-413f-48c1-aeee-2538cb0afb2f\") " Oct 30 13:12:23.137585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.137409 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-credential-socket" pod "d5a94ef0-413f-48c1-aeee-2538cb0afb2f" (UID: "d5a94ef0-413f-48c1-aeee-2538cb0afb2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:23.138227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138200 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138231 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2wh6\" (UniqueName: \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-kube-api-access-j2wh6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138246 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b711a4-1edf-4405-b7cc-b493017f2949-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138261 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138275 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138288 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138301 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138314 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b711a4-1edf-4405-b7cc-b493017f2949-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138343 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b711a4-1edf-4405-b7cc-b493017f2949-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138356 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138368 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138381 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.138868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.138395 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b711a4-1edf-4405-b7cc-b493017f2949-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.139363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.139312 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-podinfo" pod "d5a94ef0-413f-48c1-aeee-2538cb0afb2f" (UID: "d5a94ef0-413f-48c1-aeee-2538cb0afb2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:23.139448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.139374 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-envoy" pod "d5a94ef0-413f-48c1-aeee-2538cb0afb2f" (UID: "d5a94ef0-413f-48c1-aeee-2538cb0afb2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:23.139964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.139946 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-kube-api-access-plf8l" pod "d5a94ef0-413f-48c1-aeee-2538cb0afb2f" (UID: "d5a94ef0-413f-48c1-aeee-2538cb0afb2f"). InnerVolumeSpecName "kube-api-access-plf8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:23.140316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.140302 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-token" pod "d5a94ef0-413f-48c1-aeee-2538cb0afb2f" (UID: "d5a94ef0-413f-48c1-aeee-2538cb0afb2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:23.239744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.239707 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.239744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.239738 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.239744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.239746 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.239972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.239755 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plf8l\" (UniqueName: \"kubernetes.io/projected/d5a94ef0-413f-48c1-aeee-2538cb0afb2f-kube-api-access-plf8l\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.340896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.340856 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" path="/var/lib/kubelet/pods/89b711a4-1edf-4405-b7cc-b493017f2949/volumes" Oct 30 13:12:23.697508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.697414 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:23.893345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:23.892828 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:24.085071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:24.085038 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" Oct 30 13:12:24.085689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:24.085045 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl" event={"ID":"d5a94ef0-413f-48c1-aeee-2538cb0afb2f","Type":"ContainerDied","Data":"5b18419f7fadfa67edbc3cb93be4d38998aeb440ec0ef7691f716f242c0be788"} Oct 30 13:12:24.085689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:24.085166 5049 scope.go:122] "RemoveContainer" containerID="d5c207c58ba310336260f77dc060ae2bf5f0cba08ecea266c3f74400ffadf54a" Oct 30 13:12:24.233483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:24.233444 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl"] Oct 30 13:12:24.241112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:24.241084 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-911fe1d6-openshift-default-6879c75f7f-rzhgl"] Oct 30 13:12:25.297972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.297939 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:25.340698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.340659 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" path="/var/lib/kubelet/pods/d5a94ef0-413f-48c1-aeee-2538cb0afb2f/volumes" Oct 30 13:12:25.368187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.368151 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7"] Oct 30 13:12:25.368933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.368917 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" Oct 30 13:12:25.368997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.368934 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" Oct 30 13:12:25.368997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.368955 5049 cpu_manager.go:414] "containerMap: removing container" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" Oct 30 13:12:25.368997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.368961 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" Oct 30 13:12:25.368997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.368971 5049 cpu_manager.go:414] "containerMap: removing container" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" Oct 30 13:12:25.368997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.368977 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" Oct 30 13:12:25.369153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.369003 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" Oct 30 13:12:25.369153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.369008 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" Oct 30 13:12:25.369153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.369127 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" Oct 30 13:12:25.369153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.369138 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a94ef0-413f-48c1-aeee-2538cb0afb2f" containerName="istio-proxy" Oct 30 13:12:25.369153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.369146 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" Oct 30 13:12:25.369153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.369153 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b711a4-1edf-4405-b7cc-b493017f2949" containerName="istio-proxy" Oct 30 13:12:25.377136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.377097 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:12:25.399390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.399362 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7"] Oct 30 13:12:25.401828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.401806 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f0a31b\"" Oct 30 13:12:25.401935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.401836 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f0a31b-dockercfg-nz7qk\"" Oct 30 13:12:25.427373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.419452 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz"] Oct 30 13:12:25.427373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.425804 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:12:25.454989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.454952 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz"] Oct 30 13:12:25.562412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.531094 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh9xf\" (UniqueName: \"kubernetes.io/projected/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-kube-api-access-kh9xf\") pod \"payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7\" (UID: \"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:12:25.562412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.531287 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7\" (UID: \"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:12:25.562412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.547148 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:25.685344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.665920 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh9xf\" (UniqueName: \"kubernetes.io/projected/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-kube-api-access-kh9xf\") pod \"payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7\" (UID: \"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:12:25.689342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.688487 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccqfg\" (UniqueName: \"kubernetes.io/projected/e1fb92c3-28ac-4223-acee-0cc0b8100506-kube-api-access-ccqfg\") pod \"payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz\" (UID: \"e1fb92c3-28ac-4223-acee-0cc0b8100506\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:12:25.689342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.688623 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7\" (UID: \"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:12:25.689342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.688882 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fb92c3-28ac-4223-acee-0cc0b8100506-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz\" (UID: \"e1fb92c3-28ac-4223-acee-0cc0b8100506\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:12:25.722431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.690429 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7\" (UID: \"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:12:25.781346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.774204 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:25.799423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.791960 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccqfg\" (UniqueName: \"kubernetes.io/projected/e1fb92c3-28ac-4223-acee-0cc0b8100506-kube-api-access-ccqfg\") pod \"payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz\" (UID: \"e1fb92c3-28ac-4223-acee-0cc0b8100506\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:12:25.799423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.792273 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fb92c3-28ac-4223-acee-0cc0b8100506-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz\" (UID: \"e1fb92c3-28ac-4223-acee-0cc0b8100506\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:12:25.799423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.793355 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fb92c3-28ac-4223-acee-0cc0b8100506-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz\" (UID: \"e1fb92c3-28ac-4223-acee-0cc0b8100506\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:12:25.847507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.838527 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh9xf\" (UniqueName: \"kubernetes.io/projected/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-kube-api-access-kh9xf\") pod \"payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7\" (UID: \"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:12:25.894428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.890498 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccqfg\" (UniqueName: \"kubernetes.io/projected/e1fb92c3-28ac-4223-acee-0cc0b8100506-kube-api-access-ccqfg\") pod \"payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz\" (UID: \"e1fb92c3-28ac-4223-acee-0cc0b8100506\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:12:26.001500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:25.999675 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:12:26.059369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:26.058663 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:12:26.117428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:26.116376 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:26.271868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:26.271837 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:26.335236 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:12:26.335205 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5aa42c_f531_44af_9bd7_9e2d600d6a1c.slice/crio-ee05bb07d4026d342ad71f5ec3a8b366acca37c4c4d4ab66b3b680ce4821a1ff WatchSource:0}: Error finding container ee05bb07d4026d342ad71f5ec3a8b366acca37c4c4d4ab66b3b680ce4821a1ff: Status 404 returned error can't find the container with id ee05bb07d4026d342ad71f5ec3a8b366acca37c4c4d4ab66b3b680ce4821a1ff Oct 30 13:12:26.335723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:26.335567 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7"] Oct 30 13:12:26.398930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:26.398903 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz"] Oct 30 13:12:26.399411 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:12:26.399376 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fb92c3_28ac_4223_acee_0cc0b8100506.slice/crio-861715ce6c426936107ab027571029773f3ccaea1c2e331a2effafedac3377c0 WatchSource:0}: Error finding container 861715ce6c426936107ab027571029773f3ccaea1c2e331a2effafedac3377c0: Status 404 returned error can't find the container with id 861715ce6c426936107ab027571029773f3ccaea1c2e331a2effafedac3377c0 Oct 30 13:12:26.683563 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:12:26.683462 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:12:27.132330 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:27.132290 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" event={"ID":"e1fb92c3-28ac-4223-acee-0cc0b8100506","Type":"ContainerStarted","Data":"fbb41365b1db700892acdf302872986c99980bef4a0ec80a6fcaae2ee5751948"} Oct 30 13:12:27.132533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:27.132398 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" event={"ID":"e1fb92c3-28ac-4223-acee-0cc0b8100506","Type":"ContainerStarted","Data":"861715ce6c426936107ab027571029773f3ccaea1c2e331a2effafedac3377c0"} Oct 30 13:12:27.132731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:27.132710 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:12:27.134762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:27.134736 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" event={"ID":"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c","Type":"ContainerStarted","Data":"31c7b02ea9a9e9a1aecf62bd116e11820c94c339c0aacab39a460ccb2023857d"} Oct 30 13:12:27.134876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:27.134784 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" event={"ID":"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c","Type":"ContainerStarted","Data":"ee05bb07d4026d342ad71f5ec3a8b366acca37c4c4d4ab66b3b680ce4821a1ff"} Oct 30 13:12:27.134876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:27.134824 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:12:27.172975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:27.172926 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" podStartSLOduration=2.172911599 podStartE2EDuration="2.172911599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:27.169973963 +0000 UTC m=+1253.985168388" watchObservedRunningTime="2026-10-30 13:12:27.172911599 +0000 UTC m=+1253.988106075" Oct 30 13:12:27.210518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:27.210470 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" podStartSLOduration=2.210455612 podStartE2EDuration="2.210455612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:27.206656234 +0000 UTC m=+1254.021850658" watchObservedRunningTime="2026-10-30 13:12:27.210455612 +0000 UTC m=+1254.025650034" Oct 30 13:12:27.541173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:27.541068 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:29.422341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:29.414887 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:29.563653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:29.563620 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:29.711343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:29.711254 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:30.040013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:30.039924 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr"] Oct 30 13:12:30.047231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:30.047199 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a050ff-29fqr"] Oct 30 13:12:31.227343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:31.218988 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:31.339465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:31.339433 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd1dc37-2b68-44e9-8b82-d836ab18a7a0" path="/var/lib/kubelet/pods/5bd1dc37-2b68-44e9-8b82-d836ab18a7a0/volumes" Oct 30 13:12:33.060957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:33.060926 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:12:33.090393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:33.074851 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:33.923566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:33.923520 5049 scope.go:122] "RemoveContainer" containerID="487f155047b72d3b5c03ed52c245ca995a1337b4e63ef84a882ae813a61ad2d2" Oct 30 13:12:33.938594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:33.938568 5049 scope.go:122] "RemoveContainer" containerID="4b3a2dc53eac303cfbf0d30279d5d699f0a9e6270ee212c84dcfa69eedf9c640" Oct 30 13:12:33.954334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:33.954291 5049 scope.go:122] "RemoveContainer" containerID="02a136e1088ada466f26074e6063ed142a99bf56529b7c862b398ca5e9f4df40" Oct 30 13:12:33.967078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:33.967051 5049 scope.go:122] "RemoveContainer" containerID="cbc9d757ad24c8f35138f70181251d0f2f0f665fcbec422563a037f288c08a24" Oct 30 13:12:33.979861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:33.979831 5049 scope.go:122] "RemoveContainer" containerID="62057525f7456970587703923ac57f41084073a6412af20e11cc63bf5b8a716d" Oct 30 13:12:33.992573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:33.992545 5049 scope.go:122] "RemoveContainer" containerID="ab0f4300808c44780c318abb2ab31cecbc42d5ab1688d46e28563b1a2eb7163a" Oct 30 13:12:34.007176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.007145 5049 scope.go:122] "RemoveContainer" containerID="e936f0265a3c97df6758401097cd922f4242bc823e81bdae743e53071e8d02d8" Oct 30 13:12:34.020282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.020254 5049 scope.go:122] "RemoveContainer" containerID="54342181d6a93f22d63c90934db286db6fff25c307d0282f74b4d6604a8aa1fb" Oct 30 13:12:34.035295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.035268 5049 scope.go:122] "RemoveContainer" containerID="f0688171d6efe6017cdd18bbc44281736e278095587fb1961a159acdea9ff81e" Oct 30 13:12:34.836909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.836870 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl"] Oct 30 13:12:34.845812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.845480 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:34.877679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.877644 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl"] Oct 30 13:12:34.899309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.899183 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-df6c3661-serving-cert\"" Oct 30 13:12:34.942767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.942723 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-openshift-service-ca\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:34.943194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.943063 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-maas-api-tls\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:34.943444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.943427 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jctgz\" (UniqueName: \"kubernetes.io/projected/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-kube-api-access-jctgz\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:34.943606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.943595 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-metrics-certs\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:34.960625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:34.957891 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:35.048365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.045575 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jctgz\" (UniqueName: \"kubernetes.io/projected/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-kube-api-access-jctgz\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:35.048365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.045707 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-metrics-certs\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:35.048365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.046007 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-openshift-service-ca\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:35.048365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.046040 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-maas-api-tls\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:35.048365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.047665 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-openshift-service-ca\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:35.095548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.092333 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-metrics-certs\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:35.101548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.101515 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-maas-api-tls\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:35.124342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.112053 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jctgz\" (UniqueName: \"kubernetes.io/projected/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-kube-api-access-jctgz\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-qmmrl\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:35.128750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.128713 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:35.184976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.184941 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:35.558880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:35.558844 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl"] Oct 30 13:12:35.559514 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:12:35.559490 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2ddfba_7d68_4bf1_a89c_ae6f3977d942.slice/crio-a9c87670edf85ed9c4b5458ee09af3659c70a872b706cee00ab09eb5be812ea7 WatchSource:0}: Error finding container a9c87670edf85ed9c4b5458ee09af3659c70a872b706cee00ab09eb5be812ea7: Status 404 returned error can't find the container with id a9c87670edf85ed9c4b5458ee09af3659c70a872b706cee00ab09eb5be812ea7 Oct 30 13:12:36.347342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:36.338529 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" event={"ID":"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942","Type":"ContainerStarted","Data":"bab98d32ebc8ea4e65ee5ec2b1c8708e84a2c64f87cdaf927274e9d631c081a9"} Oct 30 13:12:36.347342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:36.338597 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" event={"ID":"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942","Type":"ContainerStarted","Data":"a9c87670edf85ed9c4b5458ee09af3659c70a872b706cee00ab09eb5be812ea7"} Oct 30 13:12:36.347342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:36.339869 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:36.435345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:36.432089 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" podStartSLOduration=2.044500542 podStartE2EDuration="2.432068199s" totalImagesPullingTime="387.567657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:36.425607818 +0000 UTC m=+1263.240802242" watchObservedRunningTime="2026-10-30 13:12:36.432068199 +0000 UTC m=+1263.247262624" Oct 30 13:12:36.981896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:36.981793 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:38.151712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:38.151676 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:12:38.152480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:38.152460 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:12:39.006415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:39.006378 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:41.109580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:41.109531 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:41.272153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:41.272111 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:41.841438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:41.841375 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj"] Oct 30 13:12:41.848430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:41.848395 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:41.861214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:41.861179 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj"] Oct 30 13:12:41.866519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:41.866496 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:41.877940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:41.876499 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:12:41.948022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:41.947453 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b39a67e-6swfj\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:41.948022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:41.947756 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkwtg\" (UniqueName: \"kubernetes.io/projected/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-kube-api-access-zkwtg\") pod \"maas-api-revoke-sub-5b39a67e-6swfj\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:41.948022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:41.947809 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b39a67e-6swfj\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:42.050140 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:42.050089 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b39a67e-6swfj\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:42.050450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:42.050429 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkwtg\" (UniqueName: \"kubernetes.io/projected/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-kube-api-access-zkwtg\") pod \"maas-api-revoke-sub-5b39a67e-6swfj\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:42.050540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:42.050489 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b39a67e-6swfj\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:42.051267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:42.051242 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b39a67e-6swfj\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:42.053501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:42.053475 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b39a67e-6swfj\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:42.072399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:42.072364 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkwtg\" (UniqueName: \"kubernetes.io/projected/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-kube-api-access-zkwtg\") pod \"maas-api-revoke-sub-5b39a67e-6swfj\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:42.188605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:42.188254 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:42.754459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:42.748696 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:42.897430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:42.885788 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj"] Oct 30 13:12:43.100240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:43.100198 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:43.380207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:43.380138 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:12:43.518092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:43.518050 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" event={"ID":"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b","Type":"ContainerStarted","Data":"5086f0af8c526e487ae7ca7b1653fcc2c5ca8d445b3c03017b02364e3edeab7b"} Oct 30 13:12:43.518248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:43.518126 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" event={"ID":"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b","Type":"ContainerStarted","Data":"8aaf7d33eec85c97f3150eb518e645122d58269829e426b37bbdee8b2fbe969e"} Oct 30 13:12:43.622399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:43.618717 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" podStartSLOduration=2.618693522 podStartE2EDuration="2.618693522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:43.554809731 +0000 UTC m=+1270.370004154" watchObservedRunningTime="2026-10-30 13:12:43.618693522 +0000 UTC m=+1270.433887946" Oct 30 13:12:44.038595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:44.038558 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26"] Oct 30 13:12:44.048410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:44.048372 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb60638f-4qm26"] Oct 30 13:12:44.531682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:44.531642 5049 generic.go:356] "Generic (PLEG): container finished" podID="2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b" containerID="5086f0af8c526e487ae7ca7b1653fcc2c5ca8d445b3c03017b02364e3edeab7b" exitCode=0 Oct 30 13:12:44.532202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:44.531728 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" event={"ID":"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b","Type":"ContainerDied","Data":"5086f0af8c526e487ae7ca7b1653fcc2c5ca8d445b3c03017b02364e3edeab7b"} Oct 30 13:12:44.897167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:44.897131 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:45.339508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.339476 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f926f34-337e-4cd1-a1af-b0c443c154cb" path="/var/lib/kubelet/pods/5f926f34-337e-4cd1-a1af-b0c443c154cb/volumes" Oct 30 13:12:45.542454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.542415 5049 generic.go:356] "Generic (PLEG): container finished" podID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerID="ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8" exitCode=0 Oct 30 13:12:45.542849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.542500 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" event={"ID":"b4bcdc21-0f01-4986-a3d4-1a81611835f4","Type":"ContainerDied","Data":"ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8"} Oct 30 13:12:45.543226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.543168 5049 scope.go:122] "RemoveContainer" containerID="ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8" Oct 30 13:12:45.545260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.545227 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:45.545389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.545367 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:45.545450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.545435 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:45.789396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.787891 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:45.882145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.882053 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-xks-ca\") pod \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " Oct 30 13:12:45.882145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.882110 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-kube-api-access-zkwtg\" (UniqueName: \"kubernetes.io/projected/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-kube-api-access-zkwtg\") pod \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " Oct 30 13:12:45.882411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.882148 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-openshift-ca\") pod \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\" (UID: \"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b\") " Oct 30 13:12:45.882662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.882633 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-openshift-ca" pod "2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b" (UID: "2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:45.883231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.883199 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:45.896396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.889917 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-kube-api-access-zkwtg" pod "2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b" (UID: "2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b"). InnerVolumeSpecName "kube-api-access-zkwtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:45.896396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.893667 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-xks-ca" pod "2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b" (UID: "2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:45.985300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.985266 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:45.985300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:45.985297 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkwtg\" (UniqueName: \"kubernetes.io/projected/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b-kube-api-access-zkwtg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:12:46.036188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.036113 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6"] Oct 30 13:12:46.050750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.050710 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857e706d-sdzv6"] Oct 30 13:12:46.538870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.538833 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4"] Oct 30 13:12:46.539674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.539658 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b" containerName="revoke-keys" Oct 30 13:12:46.539734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.539674 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b" containerName="revoke-keys" Oct 30 13:12:46.539798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.539789 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b" containerName="revoke-keys" Oct 30 13:12:46.543979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.543954 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.554748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.554723 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" event={"ID":"b4bcdc21-0f01-4986-a3d4-1a81611835f4","Type":"ContainerStarted","Data":"10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c"} Oct 30 13:12:46.556369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.556344 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" event={"ID":"2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b","Type":"ContainerDied","Data":"8aaf7d33eec85c97f3150eb518e645122d58269829e426b37bbdee8b2fbe969e"} Oct 30 13:12:46.556453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.556370 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj" Oct 30 13:12:46.556453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.556385 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aaf7d33eec85c97f3150eb518e645122d58269829e426b37bbdee8b2fbe969e" Oct 30 13:12:46.569367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.563994 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-f0a31b-serving-cert\"" Oct 30 13:12:46.571515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.571487 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4"] Oct 30 13:12:46.595988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.595954 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.596150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.596109 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.596391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.596371 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.596580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.596562 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddpg\" (UniqueName: \"kubernetes.io/projected/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-kube-api-access-lddpg\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.670247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.670190 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:46.698184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.698151 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.698399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.698312 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.698496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.698460 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddpg\" (UniqueName: \"kubernetes.io/projected/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-kube-api-access-lddpg\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.698555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.698512 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.698881 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:12:46.698724 5049 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-serving-cert: secret "maas-api-e2e-shared-a-w4-f0a31b-serving-cert" not found Oct 30 13:12:46.698881 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:12:46.698785 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-maas-api-tls podName:fff531b7-9d8c-4fa0-9cc8-744df2fc599c nodeName:}" failed. No retries permitted until 2026-10-30 13:12:47.198767935 +0000 UTC m=+1274.013962335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-maas-api-tls") pod "maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" (UID: "fff531b7-9d8c-4fa0-9cc8-744df2fc599c") : secret "maas-api-e2e-shared-a-w4-f0a31b-serving-cert" not found Oct 30 13:12:46.699048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.698972 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.701243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.701220 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.774709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.770434 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddpg\" (UniqueName: \"kubernetes.io/projected/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-kube-api-access-lddpg\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:46.840949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:46.840892 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:47.066921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.066884 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd"] Oct 30 13:12:47.073026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.072999 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb"] Oct 30 13:12:47.077955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.077933 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc60d35-sb2kb"] Oct 30 13:12:47.082501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.082480 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a838a82-7ssnd"] Oct 30 13:12:47.137433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.132229 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:47.243698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.243640 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:47.247922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.247489 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:47.347251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.347215 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31fc1d8-9c82-4ae6-b58b-9843e438e997" path="/var/lib/kubelet/pods/a31fc1d8-9c82-4ae6-b58b-9843e438e997/volumes" Oct 30 13:12:47.347830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.347805 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f88d82c2-c6c9-4568-9ed8-921a7bfc7103" path="/var/lib/kubelet/pods/f88d82c2-c6c9-4568-9ed8-921a7bfc7103/volumes" Oct 30 13:12:47.348345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.348309 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71" path="/var/lib/kubelet/pods/fb4d7f8b-dc72-4b78-ba2b-8505a18eaf71/volumes" Oct 30 13:12:47.442398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.441394 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:12:47.444352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.443223 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:12:47.444352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.443309 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 13:12:47.469355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.468851 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:47.847665 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:47.847632 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4"] Oct 30 13:12:47.848281 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:12:47.847657 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff531b7_9d8c_4fa0_9cc8_744df2fc599c.slice/crio-3ec161d4e0e56281df362d3aaeb8a91c692076a3538169a25a9b585dafc0e52c WatchSource:0}: Error finding container 3ec161d4e0e56281df362d3aaeb8a91c692076a3538169a25a9b585dafc0e52c: Status 404 returned error can't find the container with id 3ec161d4e0e56281df362d3aaeb8a91c692076a3538169a25a9b585dafc0e52c Oct 30 13:12:48.432084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:48.432037 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:12:48.432293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:48.432121 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 13:12:48.579987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:48.579957 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" event={"ID":"fff531b7-9d8c-4fa0-9cc8-744df2fc599c","Type":"ContainerStarted","Data":"b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36"} Oct 30 13:12:48.580176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:48.580005 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" event={"ID":"fff531b7-9d8c-4fa0-9cc8-744df2fc599c","Type":"ContainerStarted","Data":"3ec161d4e0e56281df362d3aaeb8a91c692076a3538169a25a9b585dafc0e52c"} Oct 30 13:12:48.580176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:48.580078 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:48.649258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:48.649201 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" podStartSLOduration=2.235888515 podStartE2EDuration="2.649183443s" totalImagesPullingTime="413.294928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:48.643682747 +0000 UTC m=+1275.458877171" watchObservedRunningTime="2026-10-30 13:12:48.649183443 +0000 UTC m=+1275.464377877" Oct 30 13:12:48.826343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:48.824667 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:49.435850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:49.435801 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:12:49.436457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:49.435860 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 13:12:50.437178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:50.436701 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:12:50.437178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:50.436770 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 13:12:50.601147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:50.601048 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.004220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:51.004167 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.384186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:51.384151 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:12:51.431870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:51.431835 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:12:51.432076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:51.431896 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 13:12:52.200750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:52.200713 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:52.443497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:52.443458 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:52.457498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:52.457407 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:12:52.472343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:52.468625 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:12:54.180790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:54.180758 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:54.596510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:54.596479 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:12:54.917204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:54.917116 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:55.403035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:55.403000 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:55.592726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:55.592690 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:56.204104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:56.204068 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:56.687579 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:12:56.687507 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:12:57.037196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:57.037113 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk"] Oct 30 13:12:57.043837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:57.043807 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d47a50f0-mfjgk"] Oct 30 13:12:57.342444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:57.342413 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f1ee5b-0838-4554-9db8-3209694fbcb6" path="/var/lib/kubelet/pods/31f1ee5b-0838-4554-9db8-3209694fbcb6/volumes" Oct 30 13:12:58.043819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:58.043779 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955"] Oct 30 13:12:58.050676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:58.050643 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-884fa692-9k955"] Oct 30 13:12:58.275630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:58.275600 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:58.482591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:58.482448 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:59.338873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:59.338839 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0536a8a-196f-4607-bf42-84e47e979236" path="/var/lib/kubelet/pods/c0536a8a-196f-4607-bf42-84e47e979236/volumes" Oct 30 13:12:59.960427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:12:59.960397 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:00.126870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:00.126832 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:00.483086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:00.483049 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:01.037375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:01.037336 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7"] Oct 30 13:13:01.049104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:01.049074 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73bcaeed-tzwn7"] Oct 30 13:13:01.342286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:01.342247 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e20f8a-c8b2-4563-a0f9-25afdc0cea37" path="/var/lib/kubelet/pods/80e20f8a-c8b2-4563-a0f9-25afdc0cea37/volumes" Oct 30 13:13:01.670444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:01.670340 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:01.870350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:01.855725 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:02.593880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:02.593846 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:04.475045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:04.474993 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:04.640888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:04.640843 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:06.699101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:06.699049 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:07.042286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:07.042204 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf"] Oct 30 13:13:07.054227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:07.054195 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2382e43-zj2lf"] Oct 30 13:13:07.448427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:07.448391 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7" path="/var/lib/kubelet/pods/f7a56c78-f6e1-4315-b2a4-a0e6e5b437b7/volumes" Oct 30 13:13:07.880864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:07.880811 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:09.529351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:09.515925 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:09.891288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:09.889498 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:09.891288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:09.889914 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:13:10.209703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.209403 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:10.234353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.230471 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:10.253083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.253037 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:10.370351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.357425 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:10.573705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.573653 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Oct 30 13:13:10.609587 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.608796 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:10.687099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.687031 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:13:10.696396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.695424 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:10.704901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.703935 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:10.891520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.891409 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 13:13:10.892347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.892301 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:13:10.893530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.891790 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:13:10.894620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:10.894601 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:13:11.286403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.279621 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:11.286403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.280135 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:13:11.286403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.282060 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:11.286403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.282509 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:13:11.409367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.398355 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:11.448219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.431929 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:13:11.625473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.625429 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:11.652586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.651494 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:13:11.687959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.687918 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:11.689228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.689203 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:13:11.719821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.719781 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:11.720492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.720462 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 13:13:11.733829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.721093 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:13:11.741699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.734076 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:13:11.913001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.908808 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:11.913001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:11.910335 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:12.022415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.022378 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9"] Oct 30 13:13:12.127431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.127391 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012bfe7c-zcsm9"] Oct 30 13:13:12.353476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.296501 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:12.353476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.299757 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 13:13:12.402022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.401979 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:12.577993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.570061 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:12.631608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.631413 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:12.703864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.703813 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:12.733350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.733278 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:12.781346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.733574 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 13:13:12.913523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.913420 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:12.914091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:12.914060 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Oct 30 13:13:13.298540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.298479 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:13.306344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.305401 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:13.427346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.402963 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:13.519346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.503054 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fbdcd3d-b29e-4138-bbea-b723c3377b9e" path="/var/lib/kubelet/pods/1fbdcd3d-b29e-4138-bbea-b723c3377b9e/volumes" Oct 30 13:13:13.690901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.687422 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:13.690901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.688853 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 30 13:13:13.717341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.710108 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:13.796767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.784226 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:13.796767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.784671 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:13.917346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.916632 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:13.932641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:13.932600 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:14.009530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.009421 5049 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 13:13:14.393954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.392858 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:13:14.400069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.400022 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:14.400245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.400157 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:13:14.424355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.419422 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:14.529352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.527605 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:14.689257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.689163 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:14.697096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.697065 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:13:14.714423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.712812 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 13:13:14.786957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.786918 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:14.949443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.922795 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:14.951221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:14.951182 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:15.056812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:15.056777 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:13:15.104644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:15.103789 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:13:15.105037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:15.105009 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:13:15.183347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:15.180761 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:13:15.199588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:15.199470 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:13:15.261831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:15.261800 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:13:15.481604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:15.476746 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:13:15.705070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:15.705028 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:15.949370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:15.949264 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:16.378978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:16.378920 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:16.711592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:16.711501 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:18.682095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:18.682021 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:19.674921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:19.674848 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:20.124664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:20.124611 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.007502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:21.007341 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.300574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:21.300537 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:22.038434 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:22.038395 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 13:13:22.038643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:22.038456 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 13:13:22.277923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:22.277836 5049 generic.go:356] "Generic (PLEG): container finished" podID="a2e3f959-a61b-4e80-9437-4597939fe709" containerID="27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830" exitCode=0 Oct 30 13:13:22.278095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:22.277922 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" event={"ID":"a2e3f959-a61b-4e80-9437-4597939fe709","Type":"ContainerDied","Data":"27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830"} Oct 30 13:13:22.278432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:22.278410 5049 scope.go:122] "RemoveContainer" containerID="27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830" Oct 30 13:13:22.280586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:22.280541 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:22.280685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:22.280652 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:22.280752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:22.280735 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:22.897840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:22.897809 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:23.289566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.289473 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" event={"ID":"a2e3f959-a61b-4e80-9437-4597939fe709","Type":"ContainerStarted","Data":"9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555"} Oct 30 13:13:23.291468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.291442 5049 generic.go:356] "Generic (PLEG): container finished" podID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerID="8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc" exitCode=0 Oct 30 13:13:23.291597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.291513 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" event={"ID":"dbb240b8-72ef-4031-aff9-c7ff4707a0aa","Type":"ContainerDied","Data":"8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc"} Oct 30 13:13:23.291954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.291939 5049 scope.go:122] "RemoveContainer" containerID="8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc" Oct 30 13:13:23.293654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.293625 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:23.293763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.293706 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:23.293832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.293759 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:23.463150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.463115 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:13:23.463316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.463242 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 13:13:23.463316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.463337 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 13:13:23.788111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:23.788077 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:24.062344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:24.056687 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv"] Oct 30 13:13:24.067344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:24.066938 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee525a9-ksbdv"] Oct 30 13:13:24.251343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:24.249286 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:24.315159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:24.315061 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" event={"ID":"dbb240b8-72ef-4031-aff9-c7ff4707a0aa","Type":"ContainerStarted","Data":"ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98"} Oct 30 13:13:24.460419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:24.460385 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 13:13:24.460710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:24.460681 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 13:13:24.649401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:24.649290 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:13:24.653756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:24.653428 5049 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 13:13:24.653756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:24.653490 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 13:13:25.062764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.062727 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6"] Oct 30 13:13:25.068222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.068175 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a70454-h7vv6"] Oct 30 13:13:25.238398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.231831 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:25.420419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.419508 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:25.463823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.463785 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 13:13:25.466554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.466520 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 13:13:25.480512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.467468 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d" path="/var/lib/kubelet/pods/b0cdaa62-2ef9-42ac-9469-8c1df36f8d5d/volumes" Oct 30 13:13:25.480512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.468201 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa69f85-0f28-40c9-b531-6ce96b3c074f" path="/var/lib/kubelet/pods/bfa69f85-0f28-40c9-b531-6ce96b3c074f/volumes" Oct 30 13:13:25.580093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.580051 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:25.641382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.641314 5049 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 13:13:25.641680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:25.641649 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 13:13:26.401617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:26.401583 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:26.459343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:26.458785 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 13:13:26.459343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:26.458849 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 13:13:26.650712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:26.650676 5049 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 13:13:26.651013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:26.650982 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 13:13:26.701518 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:13:26.701430 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:13:27.087428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:27.087393 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:13:27.268341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:27.267678 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:27.475424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:27.460626 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 13:13:27.475424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:27.460685 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 13:13:27.475424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:27.467898 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:27.726428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:27.722993 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:27.726428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:27.724045 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h"] Oct 30 13:13:27.833044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:27.832980 5049 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 13:13:27.833246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:27.833060 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 13:13:27.973959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:27.972807 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:13:28.017538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.016648 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-b29df4\"" Oct 30 13:13:28.017538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.016788 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-b29df4-dockercfg-cqh6f\"" Oct 30 13:13:28.045007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.044968 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h"] Oct 30 13:13:28.100788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.097207 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c02acd4-e3a0-4105-bdd7-73089832058b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h\" (UID: \"7c02acd4-e3a0-4105-bdd7-73089832058b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:13:28.100788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.097658 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4w48\" (UniqueName: \"kubernetes.io/projected/7c02acd4-e3a0-4105-bdd7-73089832058b-kube-api-access-h4w48\") pod \"payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h\" (UID: \"7c02acd4-e3a0-4105-bdd7-73089832058b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:13:28.199892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.199848 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c02acd4-e3a0-4105-bdd7-73089832058b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h\" (UID: \"7c02acd4-e3a0-4105-bdd7-73089832058b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:13:28.200127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.200109 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4w48\" (UniqueName: \"kubernetes.io/projected/7c02acd4-e3a0-4105-bdd7-73089832058b-kube-api-access-h4w48\") pod \"payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h\" (UID: \"7c02acd4-e3a0-4105-bdd7-73089832058b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:13:28.201693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.201665 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c02acd4-e3a0-4105-bdd7-73089832058b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h\" (UID: \"7c02acd4-e3a0-4105-bdd7-73089832058b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:13:28.257345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.257102 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4w48\" (UniqueName: \"kubernetes.io/projected/7c02acd4-e3a0-4105-bdd7-73089832058b-kube-api-access-h4w48\") pod \"payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h\" (UID: \"7c02acd4-e3a0-4105-bdd7-73089832058b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:13:28.392079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.392046 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:13:28.464480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.456586 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 13:13:28.464480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.456649 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 13:13:28.643755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.643657 5049 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 13:13:28.643755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.643742 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 13:13:28.822528 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:13:28.817424 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c02acd4_e3a0_4105_bdd7_73089832058b.slice/crio-ba4a1f3ff087acbbcd6fe0cf88b21859e908d1df10e44bd0d79089fde64a3d0e WatchSource:0}: Error finding container ba4a1f3ff087acbbcd6fe0cf88b21859e908d1df10e44bd0d79089fde64a3d0e: Status 404 returned error can't find the container with id ba4a1f3ff087acbbcd6fe0cf88b21859e908d1df10e44bd0d79089fde64a3d0e Oct 30 13:13:28.823909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:28.823872 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h"] Oct 30 13:13:29.444043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.444001 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2"] Oct 30 13:13:29.466344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.465697 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" event={"ID":"7c02acd4-e3a0-4105-bdd7-73089832058b","Type":"ContainerStarted","Data":"fb48f962aacbd9f736c92f0feb005b0c1bb56462e23f750e6df2730c2efc682a"} Oct 30 13:13:29.466344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.465746 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" event={"ID":"7c02acd4-e3a0-4105-bdd7-73089832058b","Type":"ContainerStarted","Data":"ba4a1f3ff087acbbcd6fe0cf88b21859e908d1df10e44bd0d79089fde64a3d0e"} Oct 30 13:13:29.466344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.465775 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2"] Oct 30 13:13:29.466344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.465801 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.466344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.465870 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:13:29.466344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.465921 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:13:29.466807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.466511 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:13:29.471371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.467297 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:13:29.480969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.479511 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fafb3f-serving-cert\"" Oct 30 13:13:29.500034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.499503 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" podStartSLOduration=2.49948584 podStartE2EDuration="2.49948584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:29.496661768 +0000 UTC m=+1316.311856191" watchObservedRunningTime="2026-10-30 13:13:29.49948584 +0000 UTC m=+1316.314680265" Oct 30 13:13:29.538909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.538857 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:29.542828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.542803 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/881102f6-0d45-45eb-8cb8-ea0b591d56ef-openshift-service-ca\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.543147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.543118 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-maas-api-tls\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.543205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.543147 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-metrics-certs\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.543365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.543308 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8w7k\" (UniqueName: \"kubernetes.io/projected/881102f6-0d45-45eb-8cb8-ea0b591d56ef-kube-api-access-v8w7k\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.640194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.640149 5049 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 13:13:29.640396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.640217 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 13:13:29.644711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.644676 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/881102f6-0d45-45eb-8cb8-ea0b591d56ef-openshift-service-ca\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.645150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.644848 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-maas-api-tls\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.645150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.644882 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-metrics-certs\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.645150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.645077 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8w7k\" (UniqueName: \"kubernetes.io/projected/881102f6-0d45-45eb-8cb8-ea0b591d56ef-kube-api-access-v8w7k\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.646492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.646362 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/881102f6-0d45-45eb-8cb8-ea0b591d56ef-openshift-service-ca\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.649286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.649267 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-maas-api-tls\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.651286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.651267 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-metrics-certs\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.669593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.669557 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8w7k\" (UniqueName: \"kubernetes.io/projected/881102f6-0d45-45eb-8cb8-ea0b591d56ef-kube-api-access-v8w7k\") pod \"maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:29.725402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.710665 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:29.799448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:29.796218 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:30.179359 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:13:30.179313 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881102f6_0d45_45eb_8cb8_ea0b591d56ef.slice/crio-75f4ba8d4d60dea85fa7bbfc6b4a4c67fc61f6434c9abdb36cb04d8b5e957fbe WatchSource:0}: Error finding container 75f4ba8d4d60dea85fa7bbfc6b4a4c67fc61f6434c9abdb36cb04d8b5e957fbe: Status 404 returned error can't find the container with id 75f4ba8d4d60dea85fa7bbfc6b4a4c67fc61f6434c9abdb36cb04d8b5e957fbe Oct 30 13:13:30.179865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:30.179842 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2"] Oct 30 13:13:30.470975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:30.470891 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" event={"ID":"881102f6-0d45-45eb-8cb8-ea0b591d56ef","Type":"ContainerStarted","Data":"75f4ba8d4d60dea85fa7bbfc6b4a4c67fc61f6434c9abdb36cb04d8b5e957fbe"} Oct 30 13:13:30.644527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:30.644502 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:13:30.645715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:30.645697 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:13:31.421348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:31.420821 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:31.493504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:31.493451 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" event={"ID":"881102f6-0d45-45eb-8cb8-ea0b591d56ef","Type":"ContainerStarted","Data":"8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566"} Oct 30 13:13:31.495119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:31.495094 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:31.542864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:31.542797 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" podStartSLOduration=2.178812641 podStartE2EDuration="2.542780626s" totalImagesPullingTime="363.967985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:31.538493356 +0000 UTC m=+1318.353687782" watchObservedRunningTime="2026-10-30 13:13:31.542780626 +0000 UTC m=+1318.357975049" Oct 30 13:13:33.322547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:33.322505 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:34.152259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:34.152217 5049 scope.go:122] "RemoveContainer" containerID="646019a2a0e075f9d7a732e9978408f5eb795733fec217be29996e64d0f42395" Oct 30 13:13:34.165165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:34.165137 5049 scope.go:122] "RemoveContainer" containerID="66711922c01b23a4eb0fc60e6b80d1539c46a9fc6793783f80d450245bc9addc" Oct 30 13:13:35.147782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:35.147055 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:36.957502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:36.957460 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:38.035109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:38.035071 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm"] Oct 30 13:13:38.044239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:38.044210 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-905c6e55-p4pmm"] Oct 30 13:13:38.519992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:38.519961 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:13:38.787660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:38.787563 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:39.343096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:39.343062 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5216becf-215f-4045-a425-3bca932690f1" path="/var/lib/kubelet/pods/5216becf-215f-4045-a425-3bca932690f1/volumes" Oct 30 13:13:40.258890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:40.258853 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:40.570966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:40.570929 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:40.806859 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:40.806827 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:41.101341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:41.100332 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj"] Oct 30 13:13:41.155843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:41.155577 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-027f93be-mjxsj"] Oct 30 13:13:41.341630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:41.341599 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef" path="/var/lib/kubelet/pods/6d3dfa6d-8795-44c8-8b2e-1ccfc63021ef/volumes" Oct 30 13:13:41.486457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:41.486306 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:41.499197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:41.499169 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:13:42.526707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:42.526670 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:44.799341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:44.799029 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:44.988853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:44.988818 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:45.182989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:45.182900 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:46.543104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.543060 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:46.607644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.607597 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4"] Oct 30 13:13:46.615616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.615590 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:13:46.634430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.634399 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4"] Oct 30 13:13:46.637227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.637205 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-659d72\"" Oct 30 13:13:46.654337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.654297 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-659d72-dockercfg-jfq2k\"" Oct 30 13:13:46.664797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.664763 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpqpg\" (UniqueName: \"kubernetes.io/projected/d630a793-11e7-4767-9b82-aa7725d12d14-kube-api-access-wpqpg\") pod \"payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4\" (UID: \"d630a793-11e7-4767-9b82-aa7725d12d14\") " pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:13:46.665110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.665086 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d630a793-11e7-4767-9b82-aa7725d12d14-plugins-config-volume\") pod \"payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4\" (UID: \"d630a793-11e7-4767-9b82-aa7725d12d14\") " pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:13:46.707368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.707276 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:46.886612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.886575 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpqpg\" (UniqueName: \"kubernetes.io/projected/d630a793-11e7-4767-9b82-aa7725d12d14-kube-api-access-wpqpg\") pod \"payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4\" (UID: \"d630a793-11e7-4767-9b82-aa7725d12d14\") " pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:13:46.888029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.887997 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d630a793-11e7-4767-9b82-aa7725d12d14-plugins-config-volume\") pod \"payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4\" (UID: \"d630a793-11e7-4767-9b82-aa7725d12d14\") " pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:13:46.891896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.891871 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d630a793-11e7-4767-9b82-aa7725d12d14-plugins-config-volume\") pod \"payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4\" (UID: \"d630a793-11e7-4767-9b82-aa7725d12d14\") " pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:13:46.927226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:46.927191 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:47.082828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:47.082791 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpqpg\" (UniqueName: \"kubernetes.io/projected/d630a793-11e7-4767-9b82-aa7725d12d14-kube-api-access-wpqpg\") pod \"payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4\" (UID: \"d630a793-11e7-4767-9b82-aa7725d12d14\") " pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:13:47.236030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:47.235787 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:13:47.573614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:47.573580 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4"] Oct 30 13:13:47.574756 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:13:47.574724 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd630a793_11e7_4767_9b82_aa7725d12d14.slice/crio-749a93b9bafa16512fc243a45ef7bec9551a23e4fdcab26c711eb33313d20b46 WatchSource:0}: Error finding container 749a93b9bafa16512fc243a45ef7bec9551a23e4fdcab26c711eb33313d20b46: Status 404 returned error can't find the container with id 749a93b9bafa16512fc243a45ef7bec9551a23e4fdcab26c711eb33313d20b46 Oct 30 13:13:47.889736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:47.889643 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" event={"ID":"d630a793-11e7-4767-9b82-aa7725d12d14","Type":"ContainerStarted","Data":"e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6"} Oct 30 13:13:47.889736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:47.889712 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" event={"ID":"d630a793-11e7-4767-9b82-aa7725d12d14","Type":"ContainerStarted","Data":"749a93b9bafa16512fc243a45ef7bec9551a23e4fdcab26c711eb33313d20b46"} Oct 30 13:13:47.889736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:47.889737 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:13:47.937535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:47.937472 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" podStartSLOduration=1.9374531510000002 podStartE2EDuration="1.937453151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:47.928631989 +0000 UTC m=+1334.743826412" watchObservedRunningTime="2026-10-30 13:13:47.937453151 +0000 UTC m=+1334.752647575" Oct 30 13:13:48.841347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:48.831343 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:50.594375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:50.593116 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:52.476176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:52.476134 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:52.733791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:52.733712 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:52.926675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:52.926639 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:53.046652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:53.046625 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p"] Oct 30 13:13:53.054209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:53.054175 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77dbc3d-jj26p"] Oct 30 13:13:53.341473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:53.341393 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040c860d-6c32-4fbc-ba13-56cf5a84ab5c" path="/var/lib/kubelet/pods/040c860d-6c32-4fbc-ba13-56cf5a84ab5c/volumes" Oct 30 13:13:54.484096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:54.484048 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:56.275373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:56.271188 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:56.678768 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:13:56.678729 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:13:58.296344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:58.282808 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:58.905368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:58.905312 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:13:59.032370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:59.032339 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72"] Oct 30 13:13:59.037358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:59.037333 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7dcd08-llg72"] Oct 30 13:13:59.339035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:13:59.339002 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60a574d-b990-4931-9b15-f0fc5741aa3e" path="/var/lib/kubelet/pods/a60a574d-b990-4931-9b15-f0fc5741aa3e/volumes" Oct 30 13:14:00.254188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:00.254148 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:01.178108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:01.172223 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:01.811385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:01.811309 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:02.369590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:02.369554 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:02.554340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:02.552751 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:02.723823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:02.723729 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:04.032776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:04.032743 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g"] Oct 30 13:14:04.042179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:04.042152 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d89fa0d-tpp7g"] Oct 30 13:14:04.415535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:04.415498 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:05.340989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:05.340956 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86ea149-96af-40ef-aef9-b54ec08ef356" path="/var/lib/kubelet/pods/e86ea149-96af-40ef-aef9-b54ec08ef356/volumes" Oct 30 13:14:06.297669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:06.297628 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:06.550991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:06.550902 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:06.725458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:06.725423 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:07.049737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:07.049706 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk"] Oct 30 13:14:07.059577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:07.059548 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a0264e-kjmvk"] Oct 30 13:14:07.338601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:07.338517 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7087d8e3-8fc2-4985-b20b-15c650d53368" path="/var/lib/kubelet/pods/7087d8e3-8fc2-4985-b20b-15c650d53368/volumes" Oct 30 13:14:07.944146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:07.944103 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:08.123456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:08.123420 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:09.138436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:09.137454 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:11.226400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:11.226356 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:13.158112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:13.158073 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:14.865239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:14.865201 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl"] Oct 30 13:14:14.871577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:14.871556 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:14.895812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:14.895780 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl"] Oct 30 13:14:14.899499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:14.899474 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-659d72-serving-cert\"" Oct 30 13:14:15.002741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.002693 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:15.011625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.011595 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-metrics-certs\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.011767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.011702 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58aba26d-82d5-4d0b-8d39-b4109b823511-openshift-service-ca\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.011827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.011776 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-maas-api-tls\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.012092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.012064 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lx2\" (UniqueName: \"kubernetes.io/projected/58aba26d-82d5-4d0b-8d39-b4109b823511-kube-api-access-f8lx2\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.146400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.132345 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lx2\" (UniqueName: \"kubernetes.io/projected/58aba26d-82d5-4d0b-8d39-b4109b823511-kube-api-access-f8lx2\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.146880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.146848 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-metrics-certs\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.147667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.147641 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58aba26d-82d5-4d0b-8d39-b4109b823511-openshift-service-ca\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.158153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.158117 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-maas-api-tls\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.158995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.148572 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58aba26d-82d5-4d0b-8d39-b4109b823511-openshift-service-ca\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.168391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.168277 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:15.170200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.170173 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lx2\" (UniqueName: \"kubernetes.io/projected/58aba26d-82d5-4d0b-8d39-b4109b823511-kube-api-access-f8lx2\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.171801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.171780 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-metrics-certs\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.196837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.196736 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-maas-api-tls\") pod \"maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.486364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.486248 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:15.846396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:15.844896 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl"] Oct 30 13:14:16.415052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:16.415008 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" event={"ID":"58aba26d-82d5-4d0b-8d39-b4109b823511","Type":"ContainerStarted","Data":"5ddd04974b2dd0eab5d4ce3350d44383aead7409b09c31fa299249a8d3a025b4"} Oct 30 13:14:17.045838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:17.045806 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn"] Oct 30 13:14:17.059297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:17.059269 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d7e814-wcppn"] Oct 30 13:14:17.256345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:17.256202 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:17.353995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:17.353905 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c1c19d-c997-4399-9f2c-75092707fc86" path="/var/lib/kubelet/pods/b2c1c19d-c997-4399-9f2c-75092707fc86/volumes" Oct 30 13:14:17.429079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:17.429033 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" event={"ID":"58aba26d-82d5-4d0b-8d39-b4109b823511","Type":"ContainerStarted","Data":"7c8fe8e39d3ac4994330335adcfaa3732e7496c36b1bf89746befa917e5be1a9"} Oct 30 13:14:17.429598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:17.429134 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:17.477034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:17.476970 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" podStartSLOduration=2.998988582 podStartE2EDuration="3.476952299s" totalImagesPullingTime="477.963717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:17.470556656 +0000 UTC m=+1364.285751058" watchObservedRunningTime="2026-10-30 13:14:17.476952299 +0000 UTC m=+1364.292146722" Oct 30 13:14:19.058464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:19.058430 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:20.931402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:20.931287 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:21.525294 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.525255 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:21.891462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.891424 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2"] Oct 30 13:14:21.905214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.905183 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:21.948958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.948925 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9fcb69-openshift-default-dockercfg-tt8b2\"" Oct 30 13:14:21.951214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.951186 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2"] Oct 30 13:14:21.955897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.955868 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-token\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:21.956036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.956001 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-envoy\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:21.956166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.956151 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-socket\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:21.956226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.956198 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istiod-ca-cert\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:21.956686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.956435 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-data\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:21.956686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.956614 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-certs\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:21.956686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.956640 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-podinfo\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:21.956686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.956667 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt2vn\" (UniqueName: \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-kube-api-access-lt2vn\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:21.956922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:21.956786 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-credential-socket\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.022987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.022950 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:22.057778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.057745 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-certs\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.057933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.057785 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-podinfo\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.057933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.057809 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt2vn\" (UniqueName: \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-kube-api-access-lt2vn\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.057933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.057828 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-credential-socket\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.057933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.057871 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-token\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.058206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.057936 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-envoy\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.058206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.058044 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-socket\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.058206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.058085 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istiod-ca-cert\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.058206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.058163 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-certs\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.058461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.058224 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-credential-socket\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.058461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.058251 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-data\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.058576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.058534 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-socket\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.058639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.058589 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-data\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.058916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.058893 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istiod-ca-cert\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.060362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.060340 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-envoy\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.060599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.060579 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-podinfo\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.142264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.132699 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt2vn\" (UniqueName: \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-kube-api-access-lt2vn\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.157012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.143184 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-token\") pod \"e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.189986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.189778 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:22.244254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.243039 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:22.672070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.672027 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:22.882870 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:14:22.882820 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3924f6fb_a4c9_4bf7_afad_e06fe378b1b8.slice/crio-a1a17751e0e98b2c3f53cfe52f676f49e8d828b5a019693e6b90613f90a388fa WatchSource:0}: Error finding container a1a17751e0e98b2c3f53cfe52f676f49e8d828b5a019693e6b90613f90a388fa: Status 404 returned error can't find the container with id a1a17751e0e98b2c3f53cfe52f676f49e8d828b5a019693e6b90613f90a388fa Oct 30 13:14:22.889868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.889824 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:22.889990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.889949 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:22.890054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.889994 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:22.898684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:22.898488 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2"] Oct 30 13:14:23.208720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:23.208645 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:23.473252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:23.473158 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:23.529341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:23.527466 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:14:23.662850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:23.662810 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" event={"ID":"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8","Type":"ContainerStarted","Data":"3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8"} Oct 30 13:14:23.663422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:23.663392 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" event={"ID":"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8","Type":"ContainerStarted","Data":"a1a17751e0e98b2c3f53cfe52f676f49e8d828b5a019693e6b90613f90a388fa"} Oct 30 13:14:23.725390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:23.725292 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:24.243628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:24.243586 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:24.250623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:24.250597 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:24.297698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:24.297643 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" podStartSLOduration=3.297628473 podStartE2EDuration="3.297628473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:23.71459202 +0000 UTC m=+1370.529786454" watchObservedRunningTime="2026-10-30 13:14:24.297628473 +0000 UTC m=+1371.112822896" Oct 30 13:14:24.301714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:24.301677 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:24.673170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:24.673132 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:24.674169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:24.674147 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:14:25.679374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:25.679299 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:26.347390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:26.347305 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:26.530277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:26.530242 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:26.679362 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:14:26.679247 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:14:27.290046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:27.290009 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:27.465398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:27.465284 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:27.838342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:27.835310 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:28.712988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:28.712950 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:28.820594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:28.820560 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v"] Oct 30 13:14:28.827314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:28.827289 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:14:28.845922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:28.845897 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v"] Oct 30 13:14:28.872666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:28.872629 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:28.895184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:28.895153 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9fcb69-dockercfg-tm5ns\"" Oct 30 13:14:28.895789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:28.895738 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9fcb69\"" Oct 30 13:14:28.993400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:28.977696 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af82fe26-4c25-478d-b576-8d7740721735-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v\" (UID: \"af82fe26-4c25-478d-b576-8d7740721735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:14:28.993400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:28.977757 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfqj\" (UniqueName: \"kubernetes.io/projected/af82fe26-4c25-478d-b576-8d7740721735-kube-api-access-6vfqj\") pod \"payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v\" (UID: \"af82fe26-4c25-478d-b576-8d7740721735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:14:29.128420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:29.115869 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af82fe26-4c25-478d-b576-8d7740721735-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v\" (UID: \"af82fe26-4c25-478d-b576-8d7740721735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:14:29.128420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:29.116012 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfqj\" (UniqueName: \"kubernetes.io/projected/af82fe26-4c25-478d-b576-8d7740721735-kube-api-access-6vfqj\") pod \"payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v\" (UID: \"af82fe26-4c25-478d-b576-8d7740721735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:14:29.128420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:29.124601 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af82fe26-4c25-478d-b576-8d7740721735-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v\" (UID: \"af82fe26-4c25-478d-b576-8d7740721735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:14:29.150797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:29.150754 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g"] Oct 30 13:14:29.152059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:29.152002 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfqj\" (UniqueName: \"kubernetes.io/projected/af82fe26-4c25-478d-b576-8d7740721735-kube-api-access-6vfqj\") pod \"payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v\" (UID: \"af82fe26-4c25-478d-b576-8d7740721735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:14:29.172183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:29.172133 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa9e2f8-rsl6g"] Oct 30 13:14:29.357137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:29.357097 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27f0e1a-01a0-4631-9d25-e058b3f5a90c" path="/var/lib/kubelet/pods/d27f0e1a-01a0-4631-9d25-e058b3f5a90c/volumes" Oct 30 13:14:29.448746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:29.448715 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:14:29.767680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:29.767649 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v"] Oct 30 13:14:29.768097 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:14:29.768029 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf82fe26_4c25_478d_b576_8d7740721735.slice/crio-b1d7cdbb899ecf72ef0fa33fd8b72b3c49eabccb9d4624d14b463b645c46846b WatchSource:0}: Error finding container b1d7cdbb899ecf72ef0fa33fd8b72b3c49eabccb9d4624d14b463b645c46846b: Status 404 returned error can't find the container with id b1d7cdbb899ecf72ef0fa33fd8b72b3c49eabccb9d4624d14b463b645c46846b Oct 30 13:14:30.185350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:30.180369 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:30.763954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:30.763900 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" event={"ID":"af82fe26-4c25-478d-b576-8d7740721735","Type":"ContainerStarted","Data":"cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50"} Oct 30 13:14:30.764151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:30.763976 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" event={"ID":"af82fe26-4c25-478d-b576-8d7740721735","Type":"ContainerStarted","Data":"b1d7cdbb899ecf72ef0fa33fd8b72b3c49eabccb9d4624d14b463b645c46846b"} Oct 30 13:14:30.764151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:30.764034 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:14:30.830533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:30.830475 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" podStartSLOduration=2.830460886 podStartE2EDuration="2.830460886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:30.824744205 +0000 UTC m=+1377.639938629" watchObservedRunningTime="2026-10-30 13:14:30.830460886 +0000 UTC m=+1377.645655333" Oct 30 13:14:31.174464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:31.174430 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:31.360472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:31.360433 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:31.722618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:31.721555 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:32.622090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:32.622048 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:32.970359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:32.970264 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:33.456961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:33.456927 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:33.711432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:33.711277 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:34.064339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.063718 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8"] Oct 30 13:14:34.071358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.071314 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaa75ec-7h9r8"] Oct 30 13:14:34.208842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.208806 5049 scope.go:122] "RemoveContainer" containerID="40986a1e1d85c43180f9021597a2687db026652b98dcf4ac29edf8ea39fa1355" Oct 30 13:14:34.221634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.221607 5049 scope.go:122] "RemoveContainer" containerID="308e0a7468dc933c6a27db1df5de541069d101899c339687de612c080e140fe1" Oct 30 13:14:34.232832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.232808 5049 scope.go:122] "RemoveContainer" containerID="4e56b0e13d07f8394b6f1309ece3661a6def992d417e4095de6b05cba416129f" Oct 30 13:14:34.243524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.243502 5049 scope.go:122] "RemoveContainer" containerID="797d1abd0181d37b27d06e474634e68d4b0e062311872ed086996732c4d70bc5" Oct 30 13:14:34.255066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.255046 5049 scope.go:122] "RemoveContainer" containerID="b78cf5ad28b92e6fc91d6e398dfae8a4c5f1b08c10548852564b7515799a364c" Oct 30 13:14:34.265270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.265253 5049 scope.go:122] "RemoveContainer" containerID="0c6c9dac92a0b9ece5ded130ae876bd63f3a988486c8156998f5b55fabc0e106" Oct 30 13:14:34.275102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.275082 5049 scope.go:122] "RemoveContainer" containerID="8ff0ffdebe9c72fcadd274cb85c059e4878ac5d353379e2496d7bd1a348c87ab" Oct 30 13:14:34.285414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.285389 5049 scope.go:122] "RemoveContainer" containerID="b5fc6ca86221f359764e87297adad8645f86a3b8b9750360341fcace12e7f1cd" Oct 30 13:14:34.298760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.298740 5049 scope.go:122] "RemoveContainer" containerID="7de24a50ecfa00cc1b25f8a4d41cd3e875a1694c5ccda63b3c6ac3e4182228b0" Oct 30 13:14:34.310738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.310718 5049 scope.go:122] "RemoveContainer" containerID="6154cc8f60e98791ec735ba28fabf7131aa984bfb96e21b963b7b795a916b053" Oct 30 13:14:34.321270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:34.321246 5049 scope.go:122] "RemoveContainer" containerID="baa8d5ad2547552a982b60a905f26701a66b7715f9895cf6486d47d93bff77c1" Oct 30 13:14:35.018345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:35.006788 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:35.341303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:35.341267 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143a1d42-5914-4774-8dc2-50d83050efe0" path="/var/lib/kubelet/pods/143a1d42-5914-4774-8dc2-50d83050efe0/volumes" Oct 30 13:14:36.041447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:36.041408 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n"] Oct 30 13:14:36.053521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:36.053491 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4031218c-nm46n"] Oct 30 13:14:37.341148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:37.341111 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ce8cc0-d9e8-4415-8f5e-c210be33101f" path="/var/lib/kubelet/pods/e0ce8cc0-d9e8-4415-8f5e-c210be33101f/volumes" Oct 30 13:14:37.383727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:37.383692 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:37.577549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:37.577512 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:37.782708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:37.782624 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:39.351478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:39.351444 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:39.564664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:39.564629 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:39.961922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:39.961843 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:41.730678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:41.730640 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:41.822238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:41.822209 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:14:41.904834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:41.904792 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:42.057895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:42.057853 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:43.997964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:43.997921 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:44.051457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:44.051412 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs"] Oct 30 13:14:44.057179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:44.057127 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2ff16c-72nqs"] Oct 30 13:14:45.346717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:45.346677 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad1c025-580c-4307-a704-54b35c4f2bce" path="/var/lib/kubelet/pods/5ad1c025-580c-4307-a704-54b35c4f2bce/volumes" Oct 30 13:14:45.821454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:45.815188 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:46.155241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:46.155154 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:46.341821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:46.341789 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:47.035576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:47.035535 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt"] Oct 30 13:14:47.043835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:47.043807 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d776433-l25qt"] Oct 30 13:14:47.338652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:47.338622 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799ff105-9b86-46ca-89f7-a581e94e6100" path="/var/lib/kubelet/pods/799ff105-9b86-46ca-89f7-a581e94e6100/volumes" Oct 30 13:14:47.993064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:47.990004 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:49.907640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:49.907534 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:52.060818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:52.060747 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:53.976104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:53.976063 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:55.294025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:55.293983 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:56.042200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:56.039412 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:56.379343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:56.375398 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:56.681668 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:14:56.681577 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:14:57.048820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:57.048785 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:57.296355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:57.296293 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7"] Oct 30 13:14:57.429299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:57.429202 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:57.584619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:57.584588 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:57.733387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:57.733278 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:58.143338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:14:58.141120 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:00.345277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:00.334471 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:02.033400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.033360 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s"] Oct 30 13:15:02.043666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.043630 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310a6533-tw57s"] Oct 30 13:15:02.296727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.296665 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" podUID="925e38f7-fc9d-4d6c-9c00-660ebca23641" containerName="payload-pre-processing" containerID="cri-o://0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d" gracePeriod=25 Oct 30 13:15:02.446588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.446553 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:02.634303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.634213 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:02.685494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.685464 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:15:02.738721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.738682 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd"] Oct 30 13:15:02.739786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.739761 5049 cpu_manager.go:414] "containerMap: removing container" podUID="925e38f7-fc9d-4d6c-9c00-660ebca23641" containerName="payload-pre-processing" Oct 30 13:15:02.739848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.739790 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925e38f7-fc9d-4d6c-9c00-660ebca23641" containerName="payload-pre-processing" Oct 30 13:15:02.740024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.740009 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="925e38f7-fc9d-4d6c-9c00-660ebca23641" containerName="payload-pre-processing" Oct 30 13:15:02.744254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.744215 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml"] Oct 30 13:15:02.744420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.744403 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" Oct 30 13:15:02.749438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.749413 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" Oct 30 13:15:02.778448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.762593 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:15:02.778448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.777301 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd"] Oct 30 13:15:02.783945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.783212 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925e38f7-fc9d-4d6c-9c00-660ebca23641-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925e38f7-fc9d-4d6c-9c00-660ebca23641-plugins-config-volume\") pod \"925e38f7-fc9d-4d6c-9c00-660ebca23641\" (UID: \"925e38f7-fc9d-4d6c-9c00-660ebca23641\") " Oct 30 13:15:02.783945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.783589 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925e38f7-fc9d-4d6c-9c00-660ebca23641-kube-api-access-g66xr\" (UniqueName: \"kubernetes.io/projected/925e38f7-fc9d-4d6c-9c00-660ebca23641-kube-api-access-g66xr\") pod \"925e38f7-fc9d-4d6c-9c00-660ebca23641\" (UID: \"925e38f7-fc9d-4d6c-9c00-660ebca23641\") " Oct 30 13:15:02.784137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.783946 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a556e12-40f6-473c-9608-12770d2ab66b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd\" (UID: \"6a556e12-40f6-473c-9608-12770d2ab66b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" Oct 30 13:15:02.784137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.783983 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/033eca56-a80a-4b76-bc36-78090b327c9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml\" (UID: \"033eca56-a80a-4b76-bc36-78090b327c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" Oct 30 13:15:02.785955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.784279 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzz5\" (UniqueName: \"kubernetes.io/projected/6a556e12-40f6-473c-9608-12770d2ab66b-kube-api-access-rbzz5\") pod \"maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd\" (UID: \"6a556e12-40f6-473c-9608-12770d2ab66b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" Oct 30 13:15:02.785955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.784523 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbrsp\" (UniqueName: \"kubernetes.io/projected/033eca56-a80a-4b76-bc36-78090b327c9a-kube-api-access-lbrsp\") pod \"maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml\" (UID: \"033eca56-a80a-4b76-bc36-78090b327c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" Oct 30 13:15:02.785955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.785522 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925e38f7-fc9d-4d6c-9c00-660ebca23641-plugins-config-volume" pod "925e38f7-fc9d-4d6c-9c00-660ebca23641" (UID: "925e38f7-fc9d-4d6c-9c00-660ebca23641"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:02.809242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.809208 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml"] Oct 30 13:15:02.813417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.812955 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925e38f7-fc9d-4d6c-9c00-660ebca23641-kube-api-access-g66xr" pod "925e38f7-fc9d-4d6c-9c00-660ebca23641" (UID: "925e38f7-fc9d-4d6c-9c00-660ebca23641"). InnerVolumeSpecName "kube-api-access-g66xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:02.885775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.885698 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a556e12-40f6-473c-9608-12770d2ab66b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd\" (UID: \"6a556e12-40f6-473c-9608-12770d2ab66b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" Oct 30 13:15:02.885775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.885736 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/033eca56-a80a-4b76-bc36-78090b327c9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml\" (UID: \"033eca56-a80a-4b76-bc36-78090b327c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" Oct 30 13:15:02.885984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.885934 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzz5\" (UniqueName: \"kubernetes.io/projected/6a556e12-40f6-473c-9608-12770d2ab66b-kube-api-access-rbzz5\") pod \"maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd\" (UID: \"6a556e12-40f6-473c-9608-12770d2ab66b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" Oct 30 13:15:02.886076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.886063 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbrsp\" (UniqueName: \"kubernetes.io/projected/033eca56-a80a-4b76-bc36-78090b327c9a-kube-api-access-lbrsp\") pod \"maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml\" (UID: \"033eca56-a80a-4b76-bc36-78090b327c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" Oct 30 13:15:02.886168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.886146 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g66xr\" (UniqueName: \"kubernetes.io/projected/925e38f7-fc9d-4d6c-9c00-660ebca23641-kube-api-access-g66xr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:15:02.886168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.886165 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925e38f7-fc9d-4d6c-9c00-660ebca23641-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:15:02.886575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.886555 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/033eca56-a80a-4b76-bc36-78090b327c9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml\" (UID: \"033eca56-a80a-4b76-bc36-78090b327c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" Oct 30 13:15:02.886624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.886583 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a556e12-40f6-473c-9608-12770d2ab66b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd\" (UID: \"6a556e12-40f6-473c-9608-12770d2ab66b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" Oct 30 13:15:02.908273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.908242 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbrsp\" (UniqueName: \"kubernetes.io/projected/033eca56-a80a-4b76-bc36-78090b327c9a-kube-api-access-lbrsp\") pod \"maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml\" (UID: \"033eca56-a80a-4b76-bc36-78090b327c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" Oct 30 13:15:02.908444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:02.908255 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzz5\" (UniqueName: \"kubernetes.io/projected/6a556e12-40f6-473c-9608-12770d2ab66b-kube-api-access-rbzz5\") pod \"maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd\" (UID: \"6a556e12-40f6-473c-9608-12770d2ab66b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" Oct 30 13:15:03.076008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.075975 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" Oct 30 13:15:03.116962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.116923 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" Oct 30 13:15:03.354598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.351057 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d62dff5-a62d-4b68-bd98-52388bb76f92" path="/var/lib/kubelet/pods/0d62dff5-a62d-4b68-bd98-52388bb76f92/volumes" Oct 30 13:15:03.400183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.400098 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd"] Oct 30 13:15:03.552719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.552190 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" event={"ID":"6a556e12-40f6-473c-9608-12770d2ab66b","Type":"ContainerStarted","Data":"06f4cd9f615e6314a0ec2c5aa564d78e162fc93c220935d3bbbd423aecb80001"} Oct 30 13:15:03.555175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.555147 5049 generic.go:356] "Generic (PLEG): container finished" podID="925e38f7-fc9d-4d6c-9c00-660ebca23641" containerID="0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d" exitCode=0 Oct 30 13:15:03.555314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.555230 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" Oct 30 13:15:03.555314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.555309 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" event={"ID":"925e38f7-fc9d-4d6c-9c00-660ebca23641","Type":"ContainerDied","Data":"0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d"} Oct 30 13:15:03.555314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.555362 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7" event={"ID":"925e38f7-fc9d-4d6c-9c00-660ebca23641","Type":"ContainerDied","Data":"06b234696328dabd5e2572d14952f67c3d26320dee57e08ed1ca5fb8037a14ae"} Oct 30 13:15:03.555528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.555391 5049 scope.go:122] "RemoveContainer" containerID="0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d" Oct 30 13:15:03.559240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.559218 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml"] Oct 30 13:15:03.619086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.619059 5049 scope.go:122] "RemoveContainer" containerID="0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d" Oct 30 13:15:03.619465 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:15:03.619437 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d\": container with ID starting with 0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d not found: ID does not exist" containerID="0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d" Oct 30 13:15:03.619561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.619475 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d"} err="failed to get container status \"0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d\": rpc error: code = NotFound desc = could not find container \"0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d\": container with ID starting with 0226eafea6f5f55042d797c4d97b62ff304b649d49cf41623095c7fac77af75d not found: ID does not exist" Oct 30 13:15:03.664273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.664189 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7"] Oct 30 13:15:03.672378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:03.672348 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3778dacd-677b6f4c-7zmk7"] Oct 30 13:15:04.048816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:04.048777 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs"] Oct 30 13:15:04.057686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:04.057646 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb85511-ggbgs"] Oct 30 13:15:04.565912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:04.565879 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" event={"ID":"033eca56-a80a-4b76-bc36-78090b327c9a","Type":"ContainerStarted","Data":"757d51229e08bb8218e04e15556a15b7e8550ec89603047a69f3b898f68ebd24"} Oct 30 13:15:04.566395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:04.565929 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" event={"ID":"033eca56-a80a-4b76-bc36-78090b327c9a","Type":"ContainerStarted","Data":"4d0dd84cc3142cf35867423a16a7524dc094b9e813194c7972985c75162c90dc"} Oct 30 13:15:04.567770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:04.567743 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" event={"ID":"6a556e12-40f6-473c-9608-12770d2ab66b","Type":"ContainerStarted","Data":"03314974696a4d02c8a663d97951052af4cfe6de0dc1a76f3ffee835328d49c6"} Oct 30 13:15:04.608559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:04.608492 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" podStartSLOduration=4.608474858 podStartE2EDuration="4.608474858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.601942413 +0000 UTC m=+1411.417136836" watchObservedRunningTime="2026-10-30 13:15:04.608474858 +0000 UTC m=+1411.423669283" Oct 30 13:15:04.644259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:04.644192 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" podStartSLOduration=4.644179063 podStartE2EDuration="4.644179063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.640476021 +0000 UTC m=+1411.455670443" watchObservedRunningTime="2026-10-30 13:15:04.644179063 +0000 UTC m=+1411.459373486" Oct 30 13:15:04.958002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:04.950960 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:05.340631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:05.340589 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80391d09-653e-4056-aa8d-05768237b1c8" path="/var/lib/kubelet/pods/80391d09-653e-4056-aa8d-05768237b1c8/volumes" Oct 30 13:15:05.341223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:05.341200 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925e38f7-fc9d-4d6c-9c00-660ebca23641" path="/var/lib/kubelet/pods/925e38f7-fc9d-4d6c-9c00-660ebca23641/volumes" Oct 30 13:15:05.578991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:05.578957 5049 generic.go:356] "Generic (PLEG): container finished" podID="033eca56-a80a-4b76-bc36-78090b327c9a" containerID="757d51229e08bb8218e04e15556a15b7e8550ec89603047a69f3b898f68ebd24" exitCode=60 Oct 30 13:15:05.579456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:05.579044 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" event={"ID":"033eca56-a80a-4b76-bc36-78090b327c9a","Type":"ContainerDied","Data":"757d51229e08bb8218e04e15556a15b7e8550ec89603047a69f3b898f68ebd24"} Oct 30 13:15:05.579456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:05.579386 5049 scope.go:122] "RemoveContainer" containerID="757d51229e08bb8218e04e15556a15b7e8550ec89603047a69f3b898f68ebd24" Oct 30 13:15:05.580910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:05.580889 5049 generic.go:356] "Generic (PLEG): container finished" podID="6a556e12-40f6-473c-9608-12770d2ab66b" containerID="03314974696a4d02c8a663d97951052af4cfe6de0dc1a76f3ffee835328d49c6" exitCode=60 Oct 30 13:15:05.580993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:05.580957 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" event={"ID":"6a556e12-40f6-473c-9608-12770d2ab66b","Type":"ContainerDied","Data":"03314974696a4d02c8a663d97951052af4cfe6de0dc1a76f3ffee835328d49c6"} Oct 30 13:15:05.581195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:05.581168 5049 scope.go:122] "RemoveContainer" containerID="03314974696a4d02c8a663d97951052af4cfe6de0dc1a76f3ffee835328d49c6" Oct 30 13:15:05.972904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:05.972811 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:06.123081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:06.123047 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:06.592065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:06.592031 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" event={"ID":"033eca56-a80a-4b76-bc36-78090b327c9a","Type":"ContainerStarted","Data":"cbd129757f27c4f0437e8073a41e5abaa0fc47a7bcbe5ed308ef455ecc874320"} Oct 30 13:15:06.594111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:06.594086 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" event={"ID":"6a556e12-40f6-473c-9608-12770d2ab66b","Type":"ContainerStarted","Data":"1b6396811712668c122cc4e1cddc64e38950afcb74ac00ff49ddf6755dcfd134"} Oct 30 13:15:07.073181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.073148 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:07.233895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.233848 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:07.426892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.426809 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:07.605444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.605406 5049 generic.go:356] "Generic (PLEG): container finished" podID="033eca56-a80a-4b76-bc36-78090b327c9a" containerID="cbd129757f27c4f0437e8073a41e5abaa0fc47a7bcbe5ed308ef455ecc874320" exitCode=60 Oct 30 13:15:07.605920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.605447 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" event={"ID":"033eca56-a80a-4b76-bc36-78090b327c9a","Type":"ContainerDied","Data":"cbd129757f27c4f0437e8073a41e5abaa0fc47a7bcbe5ed308ef455ecc874320"} Oct 30 13:15:07.605920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.605508 5049 scope.go:122] "RemoveContainer" containerID="757d51229e08bb8218e04e15556a15b7e8550ec89603047a69f3b898f68ebd24" Oct 30 13:15:07.605920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.605865 5049 scope.go:122] "RemoveContainer" containerID="cbd129757f27c4f0437e8073a41e5abaa0fc47a7bcbe5ed308ef455ecc874320" Oct 30 13:15:07.606303 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:15:07.606271 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml_odh-ai-gateway-infra(033eca56-a80a-4b76-bc36-78090b327c9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" Oct 30 13:15:07.607768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.607749 5049 generic.go:356] "Generic (PLEG): container finished" podID="6a556e12-40f6-473c-9608-12770d2ab66b" containerID="1b6396811712668c122cc4e1cddc64e38950afcb74ac00ff49ddf6755dcfd134" exitCode=60 Oct 30 13:15:07.607935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.607906 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" event={"ID":"6a556e12-40f6-473c-9608-12770d2ab66b","Type":"ContainerDied","Data":"1b6396811712668c122cc4e1cddc64e38950afcb74ac00ff49ddf6755dcfd134"} Oct 30 13:15:07.608285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.608267 5049 scope.go:122] "RemoveContainer" containerID="1b6396811712668c122cc4e1cddc64e38950afcb74ac00ff49ddf6755dcfd134" Oct 30 13:15:07.608702 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:15:07.608679 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd_odh-ai-gateway-infra(6a556e12-40f6-473c-9608-12770d2ab66b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" Oct 30 13:15:07.622835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.622815 5049 scope.go:122] "RemoveContainer" containerID="03314974696a4d02c8a663d97951052af4cfe6de0dc1a76f3ffee835328d49c6" Oct 30 13:15:07.901771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:07.901716 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:08.067940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:08.065860 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:08.673134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:08.671955 5049 scope.go:122] "RemoveContainer" containerID="cbd129757f27c4f0437e8073a41e5abaa0fc47a7bcbe5ed308ef455ecc874320" Oct 30 13:15:08.673134 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:15:08.672363 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml_odh-ai-gateway-infra(033eca56-a80a-4b76-bc36-78090b327c9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" Oct 30 13:15:08.698345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:08.677911 5049 scope.go:122] "RemoveContainer" containerID="1b6396811712668c122cc4e1cddc64e38950afcb74ac00ff49ddf6755dcfd134" Oct 30 13:15:08.698345 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:15:08.678278 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd_odh-ai-gateway-infra(6a556e12-40f6-473c-9608-12770d2ab66b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" Oct 30 13:15:09.424346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:09.423509 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:11.033511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:11.033470 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h"] Oct 30 13:15:11.043363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:11.043312 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b8d54f-cqd6h"] Oct 30 13:15:11.340705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:11.340672 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ffede2-0aec-4064-a02b-fba1d717902c" path="/var/lib/kubelet/pods/b2ffede2-0aec-4064-a02b-fba1d717902c/volumes" Oct 30 13:15:11.459353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:11.448852 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:13.036598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:13.036562 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7"] Oct 30 13:15:13.046657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:13.046625 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f44fd7-28mq7"] Oct 30 13:15:13.339928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:13.339893 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33e1339-fd95-4d60-82e8-d534ef00b56e" path="/var/lib/kubelet/pods/d33e1339-fd95-4d60-82e8-d534ef00b56e/volumes" Oct 30 13:15:13.418055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:13.418016 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:15.277449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:15.277408 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:15.430050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:15.430009 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:15.630573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:15.630516 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:17.328506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:17.328482 5049 scope.go:122] "RemoveContainer" containerID="1b6396811712668c122cc4e1cddc64e38950afcb74ac00ff49ddf6755dcfd134" Oct 30 13:15:17.455447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:17.455397 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:17.771004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:17.770971 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" event={"ID":"6a556e12-40f6-473c-9608-12770d2ab66b","Type":"ContainerStarted","Data":"52c52e8b54f7401f246739c38077999c03c1a2855ebf0c08dae29bae4a9bfddf"} Oct 30 13:15:18.328198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:18.328171 5049 scope.go:122] "RemoveContainer" containerID="cbd129757f27c4f0437e8073a41e5abaa0fc47a7bcbe5ed308ef455ecc874320" Oct 30 13:15:18.785842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:18.785758 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" event={"ID":"033eca56-a80a-4b76-bc36-78090b327c9a","Type":"ContainerStarted","Data":"ec13e7763c644a0f27353cef6117a2bf686cfe10773e8e5e29963b763421323f"} Oct 30 13:15:18.787955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:18.787927 5049 generic.go:356] "Generic (PLEG): container finished" podID="6a556e12-40f6-473c-9608-12770d2ab66b" containerID="52c52e8b54f7401f246739c38077999c03c1a2855ebf0c08dae29bae4a9bfddf" exitCode=60 Oct 30 13:15:18.788108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:18.787959 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" event={"ID":"6a556e12-40f6-473c-9608-12770d2ab66b","Type":"ContainerDied","Data":"52c52e8b54f7401f246739c38077999c03c1a2855ebf0c08dae29bae4a9bfddf"} Oct 30 13:15:18.788108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:18.788017 5049 scope.go:122] "RemoveContainer" containerID="1b6396811712668c122cc4e1cddc64e38950afcb74ac00ff49ddf6755dcfd134" Oct 30 13:15:18.788431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:18.788413 5049 scope.go:122] "RemoveContainer" containerID="52c52e8b54f7401f246739c38077999c03c1a2855ebf0c08dae29bae4a9bfddf" Oct 30 13:15:18.788798 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:15:18.788772 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd_odh-ai-gateway-infra(6a556e12-40f6-473c-9608-12770d2ab66b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" Oct 30 13:15:18.821427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:18.821397 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd"] Oct 30 13:15:19.031629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:19.031588 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c"] Oct 30 13:15:19.039742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:19.037095 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e785762-kxk2c"] Oct 30 13:15:19.338204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:19.338175 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131e791f-b22d-455d-bfe2-e93e3fc3efb7" path="/var/lib/kubelet/pods/131e791f-b22d-455d-bfe2-e93e3fc3efb7/volumes" Oct 30 13:15:19.503402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:19.503370 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:19.802368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:19.802313 5049 generic.go:356] "Generic (PLEG): container finished" podID="033eca56-a80a-4b76-bc36-78090b327c9a" containerID="ec13e7763c644a0f27353cef6117a2bf686cfe10773e8e5e29963b763421323f" exitCode=60 Oct 30 13:15:19.802368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:19.802348 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" event={"ID":"033eca56-a80a-4b76-bc36-78090b327c9a","Type":"ContainerDied","Data":"ec13e7763c644a0f27353cef6117a2bf686cfe10773e8e5e29963b763421323f"} Oct 30 13:15:19.802911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:19.802412 5049 scope.go:122] "RemoveContainer" containerID="cbd129757f27c4f0437e8073a41e5abaa0fc47a7bcbe5ed308ef455ecc874320" Oct 30 13:15:19.802911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:19.802900 5049 scope.go:122] "RemoveContainer" containerID="ec13e7763c644a0f27353cef6117a2bf686cfe10773e8e5e29963b763421323f" Oct 30 13:15:19.803358 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:15:19.803316 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml_odh-ai-gateway-infra(033eca56-a80a-4b76-bc36-78090b327c9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" Oct 30 13:15:19.847361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:19.847312 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml"] Oct 30 13:15:20.016451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.016265 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" Oct 30 13:15:20.200453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.200368 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a556e12-40f6-473c-9608-12770d2ab66b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a556e12-40f6-473c-9608-12770d2ab66b-maas-api-service-ca\") pod \"6a556e12-40f6-473c-9608-12770d2ab66b\" (UID: \"6a556e12-40f6-473c-9608-12770d2ab66b\") " Oct 30 13:15:20.200610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.200520 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a556e12-40f6-473c-9608-12770d2ab66b-kube-api-access-rbzz5\" (UniqueName: \"kubernetes.io/projected/6a556e12-40f6-473c-9608-12770d2ab66b-kube-api-access-rbzz5\") pod \"6a556e12-40f6-473c-9608-12770d2ab66b\" (UID: \"6a556e12-40f6-473c-9608-12770d2ab66b\") " Oct 30 13:15:20.200746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.200670 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a556e12-40f6-473c-9608-12770d2ab66b-maas-api-service-ca" pod "6a556e12-40f6-473c-9608-12770d2ab66b" (UID: "6a556e12-40f6-473c-9608-12770d2ab66b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:20.201298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.201279 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a556e12-40f6-473c-9608-12770d2ab66b-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.202810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.202784 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a556e12-40f6-473c-9608-12770d2ab66b-kube-api-access-rbzz5" pod "6a556e12-40f6-473c-9608-12770d2ab66b" (UID: "6a556e12-40f6-473c-9608-12770d2ab66b"). InnerVolumeSpecName "kube-api-access-rbzz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:20.302791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.302760 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbzz5\" (UniqueName: \"kubernetes.io/projected/6a556e12-40f6-473c-9608-12770d2ab66b-kube-api-access-rbzz5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.818162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.818137 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" Oct 30 13:15:20.818616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.818163 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd" event={"ID":"6a556e12-40f6-473c-9608-12770d2ab66b","Type":"ContainerDied","Data":"06f4cd9f615e6314a0ec2c5aa564d78e162fc93c220935d3bbbd423aecb80001"} Oct 30 13:15:20.818616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.818232 5049 scope.go:122] "RemoveContainer" containerID="52c52e8b54f7401f246739c38077999c03c1a2855ebf0c08dae29bae4a9bfddf" Oct 30 13:15:20.894625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.894566 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd"] Oct 30 13:15:20.905736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:20.905704 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6c3661-29889435-7rlwd"] Oct 30 13:15:21.014106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.014079 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" Oct 30 13:15:21.126527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.126439 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033eca56-a80a-4b76-bc36-78090b327c9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/033eca56-a80a-4b76-bc36-78090b327c9a-maas-api-service-ca\") pod \"033eca56-a80a-4b76-bc36-78090b327c9a\" (UID: \"033eca56-a80a-4b76-bc36-78090b327c9a\") " Oct 30 13:15:21.126702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.126550 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033eca56-a80a-4b76-bc36-78090b327c9a-kube-api-access-lbrsp\" (UniqueName: \"kubernetes.io/projected/033eca56-a80a-4b76-bc36-78090b327c9a-kube-api-access-lbrsp\") pod \"033eca56-a80a-4b76-bc36-78090b327c9a\" (UID: \"033eca56-a80a-4b76-bc36-78090b327c9a\") " Oct 30 13:15:21.126878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.126831 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033eca56-a80a-4b76-bc36-78090b327c9a-maas-api-service-ca" pod "033eca56-a80a-4b76-bc36-78090b327c9a" (UID: "033eca56-a80a-4b76-bc36-78090b327c9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:21.127378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.127358 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/033eca56-a80a-4b76-bc36-78090b327c9a-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.129052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.129029 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033eca56-a80a-4b76-bc36-78090b327c9a-kube-api-access-lbrsp" pod "033eca56-a80a-4b76-bc36-78090b327c9a" (UID: "033eca56-a80a-4b76-bc36-78090b327c9a"). InnerVolumeSpecName "kube-api-access-lbrsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:21.229304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.229272 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbrsp\" (UniqueName: \"kubernetes.io/projected/033eca56-a80a-4b76-bc36-78090b327c9a-kube-api-access-lbrsp\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:15:21.340445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.340415 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" path="/var/lib/kubelet/pods/6a556e12-40f6-473c-9608-12770d2ab66b/volumes" Oct 30 13:15:21.478409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.474617 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:21.831104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.831074 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" Oct 30 13:15:21.831606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.831106 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml" event={"ID":"033eca56-a80a-4b76-bc36-78090b327c9a","Type":"ContainerDied","Data":"4d0dd84cc3142cf35867423a16a7524dc094b9e813194c7972985c75162c90dc"} Oct 30 13:15:21.831606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.831176 5049 scope.go:122] "RemoveContainer" containerID="ec13e7763c644a0f27353cef6117a2bf686cfe10773e8e5e29963b763421323f" Oct 30 13:15:21.882430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.882399 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml"] Oct 30 13:15:21.887620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:21.887594 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6b0881-29889435-2wwml"] Oct 30 13:15:23.170681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:23.170641 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:23.421704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:23.421620 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" path="/var/lib/kubelet/pods/033eca56-a80a-4b76-bc36-78090b327c9a/volumes" Oct 30 13:15:24.243968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:24.243925 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:24.419704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:24.419669 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:24.618828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:24.617881 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:25.038850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:25.038798 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:26.622815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:26.622776 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:26.679715 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:15:26.679667 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:15:26.822342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:26.818908 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:28.113346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:28.108252 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:28.737820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:28.737781 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:30.742243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:30.742207 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:31.911649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:31.911618 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:32.137029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:32.136985 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:32.310126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:32.310097 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:32.893706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:32.893668 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:33.214356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:33.214257 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:33.367036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:33.367006 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:34.423218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.423187 5049 scope.go:122] "RemoveContainer" containerID="ca7d80395598890947df0ee1f11e2b256a875e33a294efceeefef1925264cb0e" Oct 30 13:15:34.425706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.425669 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:34.434886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.434864 5049 scope.go:122] "RemoveContainer" containerID="bcda07c0777c88641fa74f3496730252702b091ebaad7f7990580af81f30f44d" Oct 30 13:15:34.445766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.445741 5049 scope.go:122] "RemoveContainer" containerID="10cdd6da4083b29033353c4e0e9acd5c0cfe50f11ac96e2625bf607fc53e639e" Oct 30 13:15:34.456544 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.456522 5049 scope.go:122] "RemoveContainer" containerID="41645b56790f6f831211acf0e24a78b1e66d59dc7603101ebfc699072a5ba0c8" Oct 30 13:15:34.467485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.467462 5049 scope.go:122] "RemoveContainer" containerID="25967713bca2ebb1fef62c94215b1c816382ccc6a35dae4165eee115300c111e" Oct 30 13:15:34.478356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.478301 5049 scope.go:122] "RemoveContainer" containerID="9f644e07e710a71963cd228f51c478a1698f3174f2b55970c15c1031fd3575bf" Oct 30 13:15:34.488227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.488205 5049 scope.go:122] "RemoveContainer" containerID="60f7d6cdf28a30375beaeb5181cbea8b79c81c93d570a0d9ef0e5025b79fb2ef" Oct 30 13:15:34.498286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.498266 5049 scope.go:122] "RemoveContainer" containerID="d55de42f80899cd9caf62da018acf0e5643cf737bf5dc8a11ad6520e9c4bcf76" Oct 30 13:15:34.509474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.509446 5049 scope.go:122] "RemoveContainer" containerID="2c2291114b0302a794b1fd1b7875d7ba04b270aed78a71966b54d6de73f5eb12" Oct 30 13:15:34.528183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.528158 5049 scope.go:122] "RemoveContainer" containerID="9cd1f88a46e445c22366d437d92b0e56aefa5e8c094a52f659c5e6fcb78a03a0" Oct 30 13:15:34.580017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:34.579964 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:35.004266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:35.003488 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:36.033258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:36.033209 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz"] Oct 30 13:15:36.040185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:36.040147 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20713d97-v5dxz"] Oct 30 13:15:36.807852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:36.807814 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:37.340588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:37.340556 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d" path="/var/lib/kubelet/pods/b568f4f7-53d1-49ee-9ec8-0fb9c51e3f2d/volumes" Oct 30 13:15:40.137009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:40.136963 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:40.291383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:40.291277 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:40.455366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:40.455241 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:42.660798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:42.660767 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:42.844461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:42.844346 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:43.009244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:43.009098 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:43.313461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:43.313383 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:43.760076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:43.759994 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:44.290234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:44.290193 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:44.448256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:44.448206 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:45.766498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:45.766465 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:46.745461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:46.745425 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:47.763392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:47.763362 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:48.032986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:48.032902 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f"] Oct 30 13:15:48.040592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:48.040550 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5dace6-dms2f"] Oct 30 13:15:48.174422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:48.174386 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:48.587908 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:48.587868 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:48.750104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:48.750070 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:49.341119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:49.341083 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d930d022-36f9-4f16-9c29-f114847a9d3a" path="/var/lib/kubelet/pods/d930d022-36f9-4f16-9c29-f114847a9d3a/volumes" Oct 30 13:15:50.265386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.265220 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:50.432816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.432778 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:50.471011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.470960 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79"] Oct 30 13:15:50.471990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.471961 5049 cpu_manager.go:414] "containerMap: removing container" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" containerName="cleanup" Oct 30 13:15:50.471990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.471983 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" containerName="cleanup" Oct 30 13:15:50.472229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472001 5049 cpu_manager.go:414] "containerMap: removing container" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" containerName="cleanup" Oct 30 13:15:50.472229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472007 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" containerName="cleanup" Oct 30 13:15:50.472229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472035 5049 cpu_manager.go:414] "containerMap: removing container" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" containerName="cleanup" Oct 30 13:15:50.472229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472042 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" containerName="cleanup" Oct 30 13:15:50.472229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472058 5049 cpu_manager.go:414] "containerMap: removing container" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" containerName="cleanup" Oct 30 13:15:50.472229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472063 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" containerName="cleanup" Oct 30 13:15:50.472229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472074 5049 cpu_manager.go:414] "containerMap: removing container" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" containerName="cleanup" Oct 30 13:15:50.472229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472079 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" containerName="cleanup" Oct 30 13:15:50.472229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472228 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" containerName="cleanup" Oct 30 13:15:50.472824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472242 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" containerName="cleanup" Oct 30 13:15:50.472824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472251 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" containerName="cleanup" Oct 30 13:15:50.472824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472259 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" containerName="cleanup" Oct 30 13:15:50.472824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.472267 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="033eca56-a80a-4b76-bc36-78090b327c9a" containerName="cleanup" Oct 30 13:15:50.526781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.522850 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79"] Oct 30 13:15:50.526781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.522981 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:15:50.581384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.576087 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49389730-acb3-4502-ad9d-2d8ccad6ff15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79\" (UID: \"49389730-acb3-4502-ad9d-2d8ccad6ff15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:15:50.581384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.576422 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffnt4\" (UniqueName: \"kubernetes.io/projected/49389730-acb3-4502-ad9d-2d8ccad6ff15-kube-api-access-ffnt4\") pod \"payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79\" (UID: \"49389730-acb3-4502-ad9d-2d8ccad6ff15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:15:50.588350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.584673 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-84942a\"" Oct 30 13:15:50.588350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.584789 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-84942a-dockercfg-2ph84\"" Oct 30 13:15:50.694347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.684225 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnt4\" (UniqueName: \"kubernetes.io/projected/49389730-acb3-4502-ad9d-2d8ccad6ff15-kube-api-access-ffnt4\") pod \"payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79\" (UID: \"49389730-acb3-4502-ad9d-2d8ccad6ff15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:15:50.694347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.684450 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49389730-acb3-4502-ad9d-2d8ccad6ff15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79\" (UID: \"49389730-acb3-4502-ad9d-2d8ccad6ff15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:15:50.694347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.685578 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49389730-acb3-4502-ad9d-2d8ccad6ff15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79\" (UID: \"49389730-acb3-4502-ad9d-2d8ccad6ff15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:15:50.751281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.728727 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnt4\" (UniqueName: \"kubernetes.io/projected/49389730-acb3-4502-ad9d-2d8ccad6ff15-kube-api-access-ffnt4\") pod \"payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79\" (UID: \"49389730-acb3-4502-ad9d-2d8ccad6ff15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:15:50.984220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:50.984182 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:15:51.322058 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:15:51.322025 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49389730_acb3_4502_ad9d_2d8ccad6ff15.slice/crio-741a03b8fa515153036ebf8de78e30814744703fb213a18ab58515957016d055 WatchSource:0}: Error finding container 741a03b8fa515153036ebf8de78e30814744703fb213a18ab58515957016d055: Status 404 returned error can't find the container with id 741a03b8fa515153036ebf8de78e30814744703fb213a18ab58515957016d055 Oct 30 13:15:51.322229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:51.322056 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79"] Oct 30 13:15:52.328765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.328731 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" event={"ID":"49389730-acb3-4502-ad9d-2d8ccad6ff15","Type":"ContainerStarted","Data":"a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27"} Oct 30 13:15:52.329313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.328785 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" event={"ID":"49389730-acb3-4502-ad9d-2d8ccad6ff15","Type":"ContainerStarted","Data":"741a03b8fa515153036ebf8de78e30814744703fb213a18ab58515957016d055"} Oct 30 13:15:52.329313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.328813 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:15:52.330884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.330854 5049 generic.go:356] "Generic (PLEG): container finished" podID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerID="3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8" exitCode=0 Oct 30 13:15:52.331041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.330929 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" event={"ID":"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8","Type":"ContainerDied","Data":"3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8"} Oct 30 13:15:52.331335 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.331305 5049 scope.go:122] "RemoveContainer" containerID="3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8" Oct 30 13:15:52.332974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.332947 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:52.333047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.333017 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:52.333085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.333049 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:52.385565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.378756 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" podStartSLOduration=2.378736124 podStartE2EDuration="2.378736124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:52.368147791 +0000 UTC m=+1459.183342215" watchObservedRunningTime="2026-10-30 13:15:52.378736124 +0000 UTC m=+1459.193930549" Oct 30 13:15:52.412374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:52.411216 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:53.356132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:53.356093 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" event={"ID":"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8","Type":"ContainerStarted","Data":"8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc"} Oct 30 13:15:53.640842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:53.640738 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:54.190160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:54.190128 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:54.245726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:54.245682 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:15:54.246481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:54.246456 5049 patch_prober.go:29] interesting pod/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 13:15:54.246590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:54.246513 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 13:15:54.359354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:54.346227 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:54.673526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:54.673494 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:15:54.838094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:54.838060 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:55.247094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:55.247058 5049 patch_prober.go:29] interesting pod/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 13:15:55.247451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:55.247418 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 13:15:55.894349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:55.892995 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:56.246406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:56.246216 5049 patch_prober.go:29] interesting pod/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 13:15:56.246406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:56.246274 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 13:15:56.587950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:56.587916 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:56.679562 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:15:56.679514 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:15:57.187341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:57.184408 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:57.253343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:57.252374 5049 patch_prober.go:29] interesting pod/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 13:15:57.253343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:57.252446 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 13:15:58.244067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:58.244022 5049 patch_prober.go:29] interesting pod/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 13:15:58.244513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:58.244098 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 13:15:59.317221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:59.317188 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:15:59.318944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:59.318919 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:15:59.578924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:59.578824 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:59.759508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:15:59.759473 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:00.659662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:00.659629 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:01.350346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:01.338042 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:01.511924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:01.511888 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl"] Oct 30 13:16:01.512447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:01.512412 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" podUID="4f2ddfba-7d68-4bf1-a89c-ae6f3977d942" containerName="maas-api" containerID="cri-o://bab98d32ebc8ea4e65ee5ec2b1c8708e84a2c64f87cdaf927274e9d631c081a9" gracePeriod=30 Oct 30 13:16:01.693370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:01.693268 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:01.911094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:01.911057 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:02.130511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.128837 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:02.871455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.871406 5049 generic.go:356] "Generic (PLEG): container finished" podID="4f2ddfba-7d68-4bf1-a89c-ae6f3977d942" containerID="bab98d32ebc8ea4e65ee5ec2b1c8708e84a2c64f87cdaf927274e9d631c081a9" exitCode=0 Oct 30 13:16:02.874903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.873417 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" event={"ID":"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942","Type":"ContainerDied","Data":"bab98d32ebc8ea4e65ee5ec2b1c8708e84a2c64f87cdaf927274e9d631c081a9"} Oct 30 13:16:02.874903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.874811 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" event={"ID":"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942","Type":"ContainerDied","Data":"a9c87670edf85ed9c4b5458ee09af3659c70a872b706cee00ab09eb5be812ea7"} Oct 30 13:16:02.874903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.874842 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9c87670edf85ed9c4b5458ee09af3659c70a872b706cee00ab09eb5be812ea7" Oct 30 13:16:02.877684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.877645 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:16:02.995959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.995861 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-openshift-service-ca\") pod \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " Oct 30 13:16:02.996419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.996391 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-maas-api-tls\") pod \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " Oct 30 13:16:02.996712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.996691 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-metrics-certs\") pod \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " Oct 30 13:16:02.996934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.996896 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-kube-api-access-jctgz\" (UniqueName: \"kubernetes.io/projected/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-kube-api-access-jctgz\") pod \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\" (UID: \"4f2ddfba-7d68-4bf1-a89c-ae6f3977d942\") " Oct 30 13:16:02.999049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:02.998963 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-openshift-service-ca" pod "4f2ddfba-7d68-4bf1-a89c-ae6f3977d942" (UID: "4f2ddfba-7d68-4bf1-a89c-ae6f3977d942"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:03.013827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.013796 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-kube-api-access-jctgz" pod "4f2ddfba-7d68-4bf1-a89c-ae6f3977d942" (UID: "4f2ddfba-7d68-4bf1-a89c-ae6f3977d942"). InnerVolumeSpecName "kube-api-access-jctgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:03.014415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.014388 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-maas-api-tls" pod "4f2ddfba-7d68-4bf1-a89c-ae6f3977d942" (UID: "4f2ddfba-7d68-4bf1-a89c-ae6f3977d942"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:16:03.015551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.015497 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-metrics-certs" pod "4f2ddfba-7d68-4bf1-a89c-ae6f3977d942" (UID: "4f2ddfba-7d68-4bf1-a89c-ae6f3977d942"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:16:03.098994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.098963 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:03.098994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.098993 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jctgz\" (UniqueName: \"kubernetes.io/projected/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-kube-api-access-jctgz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:03.098994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.099002 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:03.099229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.099011 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:03.131779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.131744 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:03.326502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.326467 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:03.357525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.356043 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:16:03.522693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.522520 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:03.884784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.884744 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl" Oct 30 13:16:03.940374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.940338 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl"] Oct 30 13:16:03.950671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:03.950639 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-qmmrl"] Oct 30 13:16:04.075373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.071925 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:04.461346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.460389 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:04.657344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.656816 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2"] Oct 30 13:16:04.692414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.671914 5049 cpu_manager.go:414] "containerMap: removing container" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" containerName="cleanup" Oct 30 13:16:04.692414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.671943 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" containerName="cleanup" Oct 30 13:16:04.692414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.671972 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4f2ddfba-7d68-4bf1-a89c-ae6f3977d942" containerName="maas-api" Oct 30 13:16:04.692414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.671980 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2ddfba-7d68-4bf1-a89c-ae6f3977d942" containerName="maas-api" Oct 30 13:16:04.692414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.685755 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2ddfba-7d68-4bf1-a89c-ae6f3977d942" containerName="maas-api" Oct 30 13:16:04.692414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.685843 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a556e12-40f6-473c-9608-12770d2ab66b" containerName="cleanup" Oct 30 13:16:04.701461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.701430 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2"] Oct 30 13:16:04.701766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.701747 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:16:04.717090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.717038 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-cda0d9\"" Oct 30 13:16:04.752313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.752277 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-cda0d9-dockercfg-t4lmz\"" Oct 30 13:16:04.889913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.889876 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qztr6\" (UniqueName: \"kubernetes.io/projected/3fa7700f-de5f-4302-8147-edb31d44f888-kube-api-access-qztr6\") pod \"payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2\" (UID: \"3fa7700f-de5f-4302-8147-edb31d44f888\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:16:04.890347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.890012 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fa7700f-de5f-4302-8147-edb31d44f888-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2\" (UID: \"3fa7700f-de5f-4302-8147-edb31d44f888\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:16:04.992432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.992313 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fa7700f-de5f-4302-8147-edb31d44f888-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2\" (UID: \"3fa7700f-de5f-4302-8147-edb31d44f888\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:16:04.992797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.992733 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qztr6\" (UniqueName: \"kubernetes.io/projected/3fa7700f-de5f-4302-8147-edb31d44f888-kube-api-access-qztr6\") pod \"payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2\" (UID: \"3fa7700f-de5f-4302-8147-edb31d44f888\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:16:04.993189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:04.993167 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fa7700f-de5f-4302-8147-edb31d44f888-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2\" (UID: \"3fa7700f-de5f-4302-8147-edb31d44f888\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:16:05.014430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:05.014396 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qztr6\" (UniqueName: \"kubernetes.io/projected/3fa7700f-de5f-4302-8147-edb31d44f888-kube-api-access-qztr6\") pod \"payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2\" (UID: \"3fa7700f-de5f-4302-8147-edb31d44f888\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:16:05.027944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:05.027913 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:16:05.341744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:05.341709 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2ddfba-7d68-4bf1-a89c-ae6f3977d942" path="/var/lib/kubelet/pods/4f2ddfba-7d68-4bf1-a89c-ae6f3977d942/volumes" Oct 30 13:16:05.384869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:05.384839 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2"] Oct 30 13:16:05.386377 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:16:05.386314 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa7700f_de5f_4302_8147_edb31d44f888.slice/crio-a3019bfc4bdc00e837950343b35b48f824ad1c081673d86589c0d2579d482a69 WatchSource:0}: Error finding container a3019bfc4bdc00e837950343b35b48f824ad1c081673d86589c0d2579d482a69: Status 404 returned error can't find the container with id a3019bfc4bdc00e837950343b35b48f824ad1c081673d86589c0d2579d482a69 Oct 30 13:16:05.706554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:05.706099 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:05.906235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:05.906196 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" event={"ID":"3fa7700f-de5f-4302-8147-edb31d44f888","Type":"ContainerStarted","Data":"e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6"} Oct 30 13:16:05.906648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:05.906271 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:16:05.906648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:05.906295 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" event={"ID":"3fa7700f-de5f-4302-8147-edb31d44f888","Type":"ContainerStarted","Data":"a3019bfc4bdc00e837950343b35b48f824ad1c081673d86589c0d2579d482a69"} Oct 30 13:16:05.945945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:05.945897 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" podStartSLOduration=1.945883313 podStartE2EDuration="1.945883313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:05.942719834 +0000 UTC m=+1472.757914258" watchObservedRunningTime="2026-10-30 13:16:05.945883313 +0000 UTC m=+1472.761077736" Oct 30 13:16:06.774460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:06.774420 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:06.938720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:06.938675 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:07.994020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:07.993954 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:08.038937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.038898 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z"] Oct 30 13:16:08.049205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.049169 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-187291ac-hk42z"] Oct 30 13:16:08.236849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.236820 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:08.544028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.543987 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:08.609425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.609390 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb"] Oct 30 13:16:08.614536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.614516 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:16:08.631445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.631420 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4644a1-dockercfg-z4cq9\"" Oct 30 13:16:08.641688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.641507 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4644a1\"" Oct 30 13:16:08.647806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.647774 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb"] Oct 30 13:16:08.671014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.670980 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024addbd-1f62-4f03-9c83-6b597f210983-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb\" (UID: \"024addbd-1f62-4f03-9c83-6b597f210983\") " pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:16:08.671185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.671159 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx7fc\" (UniqueName: \"kubernetes.io/projected/024addbd-1f62-4f03-9c83-6b597f210983-kube-api-access-qx7fc\") pod \"payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb\" (UID: \"024addbd-1f62-4f03-9c83-6b597f210983\") " pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:16:08.715315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.715282 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:08.785345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.782824 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx7fc\" (UniqueName: \"kubernetes.io/projected/024addbd-1f62-4f03-9c83-6b597f210983-kube-api-access-qx7fc\") pod \"payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb\" (UID: \"024addbd-1f62-4f03-9c83-6b597f210983\") " pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:16:08.785345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.783244 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024addbd-1f62-4f03-9c83-6b597f210983-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb\" (UID: \"024addbd-1f62-4f03-9c83-6b597f210983\") " pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:16:08.785345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.784279 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024addbd-1f62-4f03-9c83-6b597f210983-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb\" (UID: \"024addbd-1f62-4f03-9c83-6b597f210983\") " pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:16:08.845391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.844276 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx7fc\" (UniqueName: \"kubernetes.io/projected/024addbd-1f62-4f03-9c83-6b597f210983-kube-api-access-qx7fc\") pod \"payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb\" (UID: \"024addbd-1f62-4f03-9c83-6b597f210983\") " pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:16:08.936345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:08.931772 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:16:09.175145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:09.175051 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:09.345703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:09.344961 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24995599-c47e-4abd-a57d-a589677fe6f1" path="/var/lib/kubelet/pods/24995599-c47e-4abd-a57d-a589677fe6f1/volumes" Oct 30 13:16:09.437757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:09.437726 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb"] Oct 30 13:16:09.440617 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:16:09.440588 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024addbd_1f62_4f03_9c83_6b597f210983.slice/crio-fb2335c7d153d8c4a269d80c26335b74655d4c860d7f3bd6dd9103e2047904f4 WatchSource:0}: Error finding container fb2335c7d153d8c4a269d80c26335b74655d4c860d7f3bd6dd9103e2047904f4: Status 404 returned error can't find the container with id fb2335c7d153d8c4a269d80c26335b74655d4c860d7f3bd6dd9103e2047904f4 Oct 30 13:16:10.079781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:10.079723 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" event={"ID":"024addbd-1f62-4f03-9c83-6b597f210983","Type":"ContainerStarted","Data":"f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32"} Oct 30 13:16:10.079982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:10.079797 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" event={"ID":"024addbd-1f62-4f03-9c83-6b597f210983","Type":"ContainerStarted","Data":"fb2335c7d153d8c4a269d80c26335b74655d4c860d7f3bd6dd9103e2047904f4"} Oct 30 13:16:10.079982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:10.079847 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:16:10.158796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:10.158726 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" podStartSLOduration=2.158704824 podStartE2EDuration="2.158704824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:10.155350876 +0000 UTC m=+1476.970545298" watchObservedRunningTime="2026-10-30 13:16:10.158704824 +0000 UTC m=+1476.973899247" Oct 30 13:16:10.603312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:10.603274 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:11.267232 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:11.267190 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:11.471888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:11.471856 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:12.141661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:12.141618 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:12.393402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:12.392853 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:12.601251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:12.601209 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:14.685353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:14.685257 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:16.714449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:16.705397 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:16.918808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:16.918776 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:16:18.639951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:18.639914 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:20.362349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:20.354832 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:21.091943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:21.091911 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:16:22.178957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:22.178915 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:24.065151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:24.065111 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n"] Oct 30 13:16:24.065743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:24.065369 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" containerID="cri-o://3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738" gracePeriod=30 Oct 30 13:16:24.077094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:24.077064 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s"] Oct 30 13:16:24.077559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:24.077528 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerName="istio-proxy" containerID="cri-o://c3f39e5c331dcb049305831e8b8fee1e517f88fec7e14cfdf8c20d1f589c7d69" gracePeriod=30 Oct 30 13:16:24.111399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:24.106039 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:26.491357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:26.484966 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:26.695509 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:16:26.695423 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:16:26.819209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:26.819171 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:27.782344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:27.780133 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:27.916470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:27.916421 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:28.306515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:28.306454 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:16:28.308389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:28.307640 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:28.414925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:28.414878 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:28.697402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:28.697273 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:28.783948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:28.783907 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:28.793092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:28.792758 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:28.934344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:28.931477 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" podUID="dad6d45c-f49c-490c-906a-94b25d2b9430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:28.934344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:28.932124 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:16:28.959399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:28.935015 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:28.959399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:28.940968 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:29.356522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.356483 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:29.538537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.538255 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6" Oct 30 13:16:29.543721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.543686 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:29.706752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.706429 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:16:29.741609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.741571 5049 generic.go:356] "Generic (PLEG): container finished" podID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerID="c3f39e5c331dcb049305831e8b8fee1e517f88fec7e14cfdf8c20d1f589c7d69" exitCode=0 Oct 30 13:16:29.741896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.741871 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" event={"ID":"37f9a87d-e4cc-40b4-b079-803e61efbaef","Type":"ContainerDied","Data":"c3f39e5c331dcb049305831e8b8fee1e517f88fec7e14cfdf8c20d1f589c7d69"} Oct 30 13:16:29.741982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.741932 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" event={"ID":"37f9a87d-e4cc-40b4-b079-803e61efbaef","Type":"ContainerDied","Data":"ce747ea59a130470890df891c973632edf7e29d19b8758e8df004dfc0b7c6645"} Oct 30 13:16:29.741982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.741959 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce747ea59a130470890df891c973632edf7e29d19b8758e8df004dfc0b7c6645" Oct 30 13:16:29.743749 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.743724 5049 generic.go:356] "Generic (PLEG): container finished" podID="55408942-f891-43f8-b85f-e5c4345c23fd" containerID="3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738" exitCode=0 Oct 30 13:16:29.744472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.744416 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" Oct 30 13:16:29.744581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.744567 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" event={"ID":"55408942-f891-43f8-b85f-e5c4345c23fd","Type":"ContainerDied","Data":"3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738"} Oct 30 13:16:29.744647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.744610 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n" event={"ID":"55408942-f891-43f8-b85f-e5c4345c23fd","Type":"ContainerDied","Data":"e73f00ca5912de6542e0f070303867c6815f712eb318d34c5bc8271bd9b94a1f"} Oct 30 13:16:29.744712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.744644 5049 scope.go:122] "RemoveContainer" containerID="3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738" Oct 30 13:16:29.752860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.752814 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:16:29.762856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.762815 5049 scope.go:122] "RemoveContainer" containerID="3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738" Oct 30 13:16:29.763421 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:16:29.763396 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738\": container with ID starting with 3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738 not found: ID does not exist" containerID="3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738" Oct 30 13:16:29.763616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.763580 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738"} err="failed to get container status \"3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738\": rpc error: code = NotFound desc = could not find container \"3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738\": container with ID starting with 3de7a40d00c3f46615f70655f8587f78a408572a2b22f9802f23ff861600f738 not found: ID does not exist" Oct 30 13:16:29.845739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.845706 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-kube-api-access-d7knr\" (UniqueName: \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-kube-api-access-d7knr\") pod \"37f9a87d-e4cc-40b4-b079-803e61efbaef\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " Oct 30 13:16:29.851113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.850028 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-certs\") pod \"37f9a87d-e4cc-40b4-b079-803e61efbaef\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " Oct 30 13:16:29.851629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.851602 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-socket\") pod \"55408942-f891-43f8-b85f-e5c4345c23fd\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " Oct 30 13:16:29.852184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852162 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-istio-token\" (UniqueName: \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-istio-token\") pod \"55408942-f891-43f8-b85f-e5c4345c23fd\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " Oct 30 13:16:29.852296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852234 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-credential-socket\") pod \"55408942-f891-43f8-b85f-e5c4345c23fd\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " Oct 30 13:16:29.852442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852303 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-credential-socket\") pod \"37f9a87d-e4cc-40b4-b079-803e61efbaef\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " Oct 30 13:16:29.852442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852365 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-podinfo\") pod \"37f9a87d-e4cc-40b4-b079-803e61efbaef\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " Oct 30 13:16:29.852442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852392 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-envoy\") pod \"55408942-f891-43f8-b85f-e5c4345c23fd\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " Oct 30 13:16:29.852606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852492 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-kube-api-access-9gprh\" (UniqueName: \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-kube-api-access-9gprh\") pod \"55408942-f891-43f8-b85f-e5c4345c23fd\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " Oct 30 13:16:29.852606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852525 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-socket\") pod \"37f9a87d-e4cc-40b4-b079-803e61efbaef\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " Oct 30 13:16:29.852606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852553 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55408942-f891-43f8-b85f-e5c4345c23fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55408942-f891-43f8-b85f-e5c4345c23fd-istio-podinfo\") pod \"55408942-f891-43f8-b85f-e5c4345c23fd\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " Oct 30 13:16:29.852753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852680 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-token\" (UniqueName: \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-token\") pod \"37f9a87d-e4cc-40b4-b079-803e61efbaef\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " Oct 30 13:16:29.852806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852756 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55408942-f891-43f8-b85f-e5c4345c23fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55408942-f891-43f8-b85f-e5c4345c23fd-istiod-ca-cert\") pod \"55408942-f891-43f8-b85f-e5c4345c23fd\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " Oct 30 13:16:29.852806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852790 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f9a87d-e4cc-40b4-b079-803e61efbaef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f9a87d-e4cc-40b4-b079-803e61efbaef-istiod-ca-cert\") pod \"37f9a87d-e4cc-40b4-b079-803e61efbaef\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " Oct 30 13:16:29.852911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852860 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-data\") pod \"55408942-f891-43f8-b85f-e5c4345c23fd\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " Oct 30 13:16:29.852970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852919 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-data\") pod \"37f9a87d-e4cc-40b4-b079-803e61efbaef\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " Oct 30 13:16:29.853025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.852969 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-certs\") pod \"55408942-f891-43f8-b85f-e5c4345c23fd\" (UID: \"55408942-f891-43f8-b85f-e5c4345c23fd\") " Oct 30 13:16:29.853078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.853027 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-envoy\") pod \"37f9a87d-e4cc-40b4-b079-803e61efbaef\" (UID: \"37f9a87d-e4cc-40b4-b079-803e61efbaef\") " Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.855650 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-data" pod "55408942-f891-43f8-b85f-e5c4345c23fd" (UID: "55408942-f891-43f8-b85f-e5c4345c23fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.855680 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55408942-f891-43f8-b85f-e5c4345c23fd-istiod-ca-cert" pod "55408942-f891-43f8-b85f-e5c4345c23fd" (UID: "55408942-f891-43f8-b85f-e5c4345c23fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.855872 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-socket" pod "37f9a87d-e4cc-40b4-b079-803e61efbaef" (UID: "37f9a87d-e4cc-40b4-b079-803e61efbaef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.856247 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f9a87d-e4cc-40b4-b079-803e61efbaef-istiod-ca-cert" pod "37f9a87d-e4cc-40b4-b079-803e61efbaef" (UID: "37f9a87d-e4cc-40b4-b079-803e61efbaef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.856753 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-data" pod "37f9a87d-e4cc-40b4-b079-803e61efbaef" (UID: "37f9a87d-e4cc-40b4-b079-803e61efbaef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.856974 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-credential-socket" pod "55408942-f891-43f8-b85f-e5c4345c23fd" (UID: "55408942-f891-43f8-b85f-e5c4345c23fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.857144 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-certs" pod "55408942-f891-43f8-b85f-e5c4345c23fd" (UID: "55408942-f891-43f8-b85f-e5c4345c23fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.857457 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-certs" pod "37f9a87d-e4cc-40b4-b079-803e61efbaef" (UID: "37f9a87d-e4cc-40b4-b079-803e61efbaef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.857677 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-socket" pod "55408942-f891-43f8-b85f-e5c4345c23fd" (UID: "55408942-f891-43f8-b85f-e5c4345c23fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.858641 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.858666 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.858689 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.858699 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.858707 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55408942-f891-43f8-b85f-e5c4345c23fd-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.858715 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f9a87d-e4cc-40b4-b079-803e61efbaef-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.858727 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.858739 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.859234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.858752 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.860534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.860510 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-credential-socket" pod "37f9a87d-e4cc-40b4-b079-803e61efbaef" (UID: "37f9a87d-e4cc-40b4-b079-803e61efbaef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:29.863725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.863698 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-istio-token" pod "55408942-f891-43f8-b85f-e5c4345c23fd" (UID: "55408942-f891-43f8-b85f-e5c4345c23fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:29.863838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.863783 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-kube-api-access-9gprh" pod "55408942-f891-43f8-b85f-e5c4345c23fd" (UID: "55408942-f891-43f8-b85f-e5c4345c23fd"). InnerVolumeSpecName "kube-api-access-9gprh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:29.864529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.864503 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-kube-api-access-d7knr" pod "37f9a87d-e4cc-40b4-b079-803e61efbaef" (UID: "37f9a87d-e4cc-40b4-b079-803e61efbaef"). InnerVolumeSpecName "kube-api-access-d7knr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:29.867459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.865370 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55408942-f891-43f8-b85f-e5c4345c23fd-istio-podinfo" pod "55408942-f891-43f8-b85f-e5c4345c23fd" (UID: "55408942-f891-43f8-b85f-e5c4345c23fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:29.867459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.865603 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-token" pod "37f9a87d-e4cc-40b4-b079-803e61efbaef" (UID: "37f9a87d-e4cc-40b4-b079-803e61efbaef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:29.867459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.865609 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-podinfo" pod "37f9a87d-e4cc-40b4-b079-803e61efbaef" (UID: "37f9a87d-e4cc-40b4-b079-803e61efbaef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:29.867459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.867030 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-envoy" pod "55408942-f891-43f8-b85f-e5c4345c23fd" (UID: "55408942-f891-43f8-b85f-e5c4345c23fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:29.874218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.874190 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-envoy" pod "37f9a87d-e4cc-40b4-b079-803e61efbaef" (UID: "37f9a87d-e4cc-40b4-b079-803e61efbaef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:29.966035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.961218 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.966035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.961288 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.966035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.961308 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7knr\" (UniqueName: \"kubernetes.io/projected/37f9a87d-e4cc-40b4-b079-803e61efbaef-kube-api-access-d7knr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.966035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.961347 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.966035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.961361 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f9a87d-e4cc-40b4-b079-803e61efbaef-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.966035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.961373 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f9a87d-e4cc-40b4-b079-803e61efbaef-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.966035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.961389 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55408942-f891-43f8-b85f-e5c4345c23fd-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.966035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.961417 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gprh\" (UniqueName: \"kubernetes.io/projected/55408942-f891-43f8-b85f-e5c4345c23fd-kube-api-access-9gprh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.966035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:29.961433 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55408942-f891-43f8-b85f-e5c4345c23fd-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:30.089876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:30.089839 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n"] Oct 30 13:16:30.095101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:30.095073 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-2c24n"] Oct 30 13:16:30.756011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:30.755985 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s" Oct 30 13:16:30.814580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:30.814548 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s"] Oct 30 13:16:30.824123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:30.824095 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-6x66s"] Oct 30 13:16:31.173647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:31.173612 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:31.338909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:31.338876 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" path="/var/lib/kubelet/pods/37f9a87d-e4cc-40b4-b079-803e61efbaef/volumes" Oct 30 13:16:31.339478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:31.339464 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" path="/var/lib/kubelet/pods/55408942-f891-43f8-b85f-e5c4345c23fd/volumes" Oct 30 13:16:33.028858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:33.028826 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:33.564025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:33.563987 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:16:33.568389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:33.568368 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:16:34.664053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:34.664019 5049 scope.go:122] "RemoveContainer" containerID="434dc04da05ac3c7bbcc56be5ea106621615ebfc1bdcbc4f3e3085f429af0243" Oct 30 13:16:34.678295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:34.678256 5049 scope.go:122] "RemoveContainer" containerID="25a6255b08128dbde9df2777099fe3b9357305c681e186c635d5bf377cba8241" Oct 30 13:16:34.689085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:34.689055 5049 scope.go:122] "RemoveContainer" containerID="89877fa87696d73c205fa5343cb25595e3f4c665c577d352f5e0aa0f39fbb5f2" Oct 30 13:16:34.699939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:34.699920 5049 scope.go:122] "RemoveContainer" containerID="711abda83face991f9902306e7b4f08d6332849ca113d9e32334937ccbbf13f7" Oct 30 13:16:34.710457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:34.710435 5049 scope.go:122] "RemoveContainer" containerID="3e29155a3ca1230f415d89359022452c0dc351a996f8e4c06bacba661121ee20" Oct 30 13:16:34.720486 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:34.720466 5049 scope.go:122] "RemoveContainer" containerID="007e92f0cd4743c5720a264d00b1b1c7f22698a3ed05b1ef066246b6d368ab92" Oct 30 13:16:34.734227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:34.734204 5049 scope.go:122] "RemoveContainer" containerID="61968ad398a1af1845276a8be2335f92bc8a246dfa0c1d1e449c3d684c012f95" Oct 30 13:16:34.743694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:34.743674 5049 scope.go:122] "RemoveContainer" containerID="c3f39e5c331dcb049305831e8b8fee1e517f88fec7e14cfdf8c20d1f589c7d69" Oct 30 13:16:34.755929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:34.755908 5049 scope.go:122] "RemoveContainer" containerID="958fca6ec2af7c1dbddbc2d1d5c93de4f641e61bdfaa7891f2b348fb00f0fbf0" Oct 30 13:16:35.221135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:35.217861 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:36.743827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:36.743793 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2"] Oct 30 13:16:36.744348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:36.744033 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" podUID="881102f6-0d45-45eb-8cb8-ea0b591d56ef" containerName="maas-api" containerID="cri-o://8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566" gracePeriod=30 Oct 30 13:16:36.855089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:36.855054 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:37.318413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.315842 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:16:37.463153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.460892 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:37.480621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.480550 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881102f6-0d45-45eb-8cb8-ea0b591d56ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/881102f6-0d45-45eb-8cb8-ea0b591d56ef-openshift-service-ca\") pod \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " Oct 30 13:16:37.488825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.488783 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881102f6-0d45-45eb-8cb8-ea0b591d56ef-kube-api-access-v8w7k\" (UniqueName: \"kubernetes.io/projected/881102f6-0d45-45eb-8cb8-ea0b591d56ef-kube-api-access-v8w7k\") pod \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " Oct 30 13:16:37.489245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.489221 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-metrics-certs\") pod \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " Oct 30 13:16:37.490218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.490196 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-maas-api-tls\") pod \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\" (UID: \"881102f6-0d45-45eb-8cb8-ea0b591d56ef\") " Oct 30 13:16:37.490799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.486958 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881102f6-0d45-45eb-8cb8-ea0b591d56ef-openshift-service-ca" pod "881102f6-0d45-45eb-8cb8-ea0b591d56ef" (UID: "881102f6-0d45-45eb-8cb8-ea0b591d56ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:37.492227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.492205 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/881102f6-0d45-45eb-8cb8-ea0b591d56ef-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:37.499936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.499799 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881102f6-0d45-45eb-8cb8-ea0b591d56ef-kube-api-access-v8w7k" pod "881102f6-0d45-45eb-8cb8-ea0b591d56ef" (UID: "881102f6-0d45-45eb-8cb8-ea0b591d56ef"). InnerVolumeSpecName "kube-api-access-v8w7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:37.500384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.500292 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-maas-api-tls" pod "881102f6-0d45-45eb-8cb8-ea0b591d56ef" (UID: "881102f6-0d45-45eb-8cb8-ea0b591d56ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:16:37.501656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.501391 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-metrics-certs" pod "881102f6-0d45-45eb-8cb8-ea0b591d56ef" (UID: "881102f6-0d45-45eb-8cb8-ea0b591d56ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:16:37.600399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.599915 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8w7k\" (UniqueName: \"kubernetes.io/projected/881102f6-0d45-45eb-8cb8-ea0b591d56ef-kube-api-access-v8w7k\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:37.600399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.599957 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:37.600399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.599975 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/881102f6-0d45-45eb-8cb8-ea0b591d56ef-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:37.729889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.729481 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:37.853194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.853099 5049 generic.go:356] "Generic (PLEG): container finished" podID="881102f6-0d45-45eb-8cb8-ea0b591d56ef" containerID="8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566" exitCode=0 Oct 30 13:16:37.853194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.853159 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" Oct 30 13:16:37.853728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.853202 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" event={"ID":"881102f6-0d45-45eb-8cb8-ea0b591d56ef","Type":"ContainerDied","Data":"8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566"} Oct 30 13:16:37.853728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.853269 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2" event={"ID":"881102f6-0d45-45eb-8cb8-ea0b591d56ef","Type":"ContainerDied","Data":"75f4ba8d4d60dea85fa7bbfc6b4a4c67fc61f6434c9abdb36cb04d8b5e957fbe"} Oct 30 13:16:37.853728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.853305 5049 scope.go:122] "RemoveContainer" containerID="8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566" Oct 30 13:16:37.866627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.866605 5049 scope.go:122] "RemoveContainer" containerID="8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566" Oct 30 13:16:37.867033 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:16:37.867012 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566\": container with ID starting with 8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566 not found: ID does not exist" containerID="8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566" Oct 30 13:16:37.867129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.867042 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566"} err="failed to get container status \"8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566\": rpc error: code = NotFound desc = could not find container \"8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566\": container with ID starting with 8c51ead327149f7f1d98ba1de7fe14ef1f0cf1d93b99deacd16d09fed032d566 not found: ID does not exist" Oct 30 13:16:37.907065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.907021 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2"] Oct 30 13:16:37.910355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.910297 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fafb3f-67dcbc6496-r2pg2"] Oct 30 13:16:37.922204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:37.922143 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:38.207826 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:38.207710 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:39.345413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:39.345377 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881102f6-0d45-45eb-8cb8-ea0b591d56ef" path="/var/lib/kubelet/pods/881102f6-0d45-45eb-8cb8-ea0b591d56ef/volumes" Oct 30 13:16:39.969548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:39.969514 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4"] Oct 30 13:16:40.138492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:40.138455 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:40.161890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:40.161849 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr"] Oct 30 13:16:40.162087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:40.162067 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerName="istio-proxy" containerID="cri-o://4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88" gracePeriod=30 Oct 30 13:16:40.478343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:40.471583 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz"] Oct 30 13:16:40.605823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:40.605777 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6"] Oct 30 13:16:40.614344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:40.606060 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerName="istio-proxy" containerID="cri-o://888d95f44fbb9e2a8dd23ffaa58fa512f8f2ef512cf35845e0053160bef0b223" gracePeriod=30 Oct 30 13:16:40.632293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:40.632261 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:40.888802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:40.888733 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" containerID="cri-o://e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571" gracePeriod=30 Oct 30 13:16:43.011084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:43.011046 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:44.878671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.878641 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw"] Oct 30 13:16:44.879464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.879449 5049 cpu_manager.go:414] "containerMap: removing container" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" Oct 30 13:16:44.879513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.879465 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" Oct 30 13:16:44.879513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.879487 5049 cpu_manager.go:414] "containerMap: removing container" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerName="istio-proxy" Oct 30 13:16:44.879513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.879493 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerName="istio-proxy" Oct 30 13:16:44.879626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.879532 5049 cpu_manager.go:414] "containerMap: removing container" podUID="881102f6-0d45-45eb-8cb8-ea0b591d56ef" containerName="maas-api" Oct 30 13:16:44.879626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.879537 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881102f6-0d45-45eb-8cb8-ea0b591d56ef" containerName="maas-api" Oct 30 13:16:44.879688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.879660 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="881102f6-0d45-45eb-8cb8-ea0b591d56ef" containerName="maas-api" Oct 30 13:16:44.879688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.879672 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="55408942-f891-43f8-b85f-e5c4345c23fd" containerName="istio-proxy" Oct 30 13:16:44.879688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.879679 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f9a87d-e4cc-40b4-b079-803e61efbaef" containerName="istio-proxy" Oct 30 13:16:44.885177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.885154 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:44.911741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.911712 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-cda0d9-serving-cert\"" Oct 30 13:16:44.921055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.921028 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw"] Oct 30 13:16:44.962751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.962718 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:44.962918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.962769 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:44.963006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.962989 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6b9\" (UniqueName: \"kubernetes.io/projected/20a120e6-66d4-4643-bd63-428f163eaf8c-kube-api-access-fr6b9\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:44.963251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.963235 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a120e6-66d4-4643-bd63-428f163eaf8c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:44.970649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:44.970613 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" podUID="d630a793-11e7-4767-9b82-aa7725d12d14" containerName="payload-processing" containerID="cri-o://e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6" gracePeriod=25 Oct 30 13:16:45.064863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.064831 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a120e6-66d4-4643-bd63-428f163eaf8c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:45.065039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.064947 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:45.065039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.064972 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:45.065157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.065065 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6b9\" (UniqueName: \"kubernetes.io/projected/20a120e6-66d4-4643-bd63-428f163eaf8c-kube-api-access-fr6b9\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:45.065722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.065695 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a120e6-66d4-4643-bd63-428f163eaf8c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:45.067559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.067531 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:45.067755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.067731 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:45.124382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.124352 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6b9\" (UniqueName: \"kubernetes.io/projected/20a120e6-66d4-4643-bd63-428f163eaf8c-kube-api-access-fr6b9\") pod \"maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:45.294569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.245337 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:45.374828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.373769 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:45.424623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.424146 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:16:45.561175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.561135 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d630a793-11e7-4767-9b82-aa7725d12d14-kube-api-access-wpqpg\" (UniqueName: \"kubernetes.io/projected/d630a793-11e7-4767-9b82-aa7725d12d14-kube-api-access-wpqpg\") pod \"d630a793-11e7-4767-9b82-aa7725d12d14\" (UID: \"d630a793-11e7-4767-9b82-aa7725d12d14\") " Oct 30 13:16:45.561395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.561363 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d630a793-11e7-4767-9b82-aa7725d12d14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d630a793-11e7-4767-9b82-aa7725d12d14-plugins-config-volume\") pod \"d630a793-11e7-4767-9b82-aa7725d12d14\" (UID: \"d630a793-11e7-4767-9b82-aa7725d12d14\") " Oct 30 13:16:45.562726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.562666 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d630a793-11e7-4767-9b82-aa7725d12d14-plugins-config-volume" pod "d630a793-11e7-4767-9b82-aa7725d12d14" (UID: "d630a793-11e7-4767-9b82-aa7725d12d14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:45.567657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.567456 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d630a793-11e7-4767-9b82-aa7725d12d14-kube-api-access-wpqpg" pod "d630a793-11e7-4767-9b82-aa7725d12d14" (UID: "d630a793-11e7-4767-9b82-aa7725d12d14"). InnerVolumeSpecName "kube-api-access-wpqpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.642312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.642284 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:16:45.646416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.646302 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:45.663185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.663148 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-kube-api-access-dm5jz\" (UniqueName: \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-kube-api-access-dm5jz\") pod \"54b70acc-a40d-4065-8c58-53a6fff7d969\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " Oct 30 13:16:45.663458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.663438 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b70acc-a40d-4065-8c58-53a6fff7d969-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b70acc-a40d-4065-8c58-53a6fff7d969-istiod-ca-cert\") pod \"54b70acc-a40d-4065-8c58-53a6fff7d969\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " Oct 30 13:16:45.663552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.663500 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-token\" (UniqueName: \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-token\") pod \"54b70acc-a40d-4065-8c58-53a6fff7d969\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " Oct 30 13:16:45.663552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.663540 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-podinfo\") pod \"54b70acc-a40d-4065-8c58-53a6fff7d969\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " Oct 30 13:16:45.663682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.663577 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-data\") pod \"54b70acc-a40d-4065-8c58-53a6fff7d969\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " Oct 30 13:16:45.663682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.663615 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-envoy\") pod \"54b70acc-a40d-4065-8c58-53a6fff7d969\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " Oct 30 13:16:45.663682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.663663 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-certs\") pod \"54b70acc-a40d-4065-8c58-53a6fff7d969\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " Oct 30 13:16:45.663841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.663697 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-credential-socket\") pod \"54b70acc-a40d-4065-8c58-53a6fff7d969\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " Oct 30 13:16:45.663841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.663810 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-socket\") pod \"54b70acc-a40d-4065-8c58-53a6fff7d969\" (UID: \"54b70acc-a40d-4065-8c58-53a6fff7d969\") " Oct 30 13:16:45.663971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.663932 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b70acc-a40d-4065-8c58-53a6fff7d969-istiod-ca-cert" pod "54b70acc-a40d-4065-8c58-53a6fff7d969" (UID: "54b70acc-a40d-4065-8c58-53a6fff7d969"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:45.664776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.664749 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpqpg\" (UniqueName: \"kubernetes.io/projected/d630a793-11e7-4767-9b82-aa7725d12d14-kube-api-access-wpqpg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.664892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.664779 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b70acc-a40d-4065-8c58-53a6fff7d969-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.664892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.664800 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d630a793-11e7-4767-9b82-aa7725d12d14-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.665038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.665018 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-socket" pod "54b70acc-a40d-4065-8c58-53a6fff7d969" (UID: "54b70acc-a40d-4065-8c58-53a6fff7d969"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.665482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.665193 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-certs" pod "54b70acc-a40d-4065-8c58-53a6fff7d969" (UID: "54b70acc-a40d-4065-8c58-53a6fff7d969"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.665482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.665374 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-credential-socket" pod "54b70acc-a40d-4065-8c58-53a6fff7d969" (UID: "54b70acc-a40d-4065-8c58-53a6fff7d969"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.665482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.665421 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-data" pod "54b70acc-a40d-4065-8c58-53a6fff7d969" (UID: "54b70acc-a40d-4065-8c58-53a6fff7d969"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.666623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.666596 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-envoy" pod "54b70acc-a40d-4065-8c58-53a6fff7d969" (UID: "54b70acc-a40d-4065-8c58-53a6fff7d969"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.667394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.667368 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-kube-api-access-dm5jz" pod "54b70acc-a40d-4065-8c58-53a6fff7d969" (UID: "54b70acc-a40d-4065-8c58-53a6fff7d969"). InnerVolumeSpecName "kube-api-access-dm5jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.667511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.667488 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-podinfo" pod "54b70acc-a40d-4065-8c58-53a6fff7d969" (UID: "54b70acc-a40d-4065-8c58-53a6fff7d969"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:45.667685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.667665 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-token" pod "54b70acc-a40d-4065-8c58-53a6fff7d969" (UID: "54b70acc-a40d-4065-8c58-53a6fff7d969"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.737589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.732688 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw"] Oct 30 13:16:45.760565 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:16:45.760522 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a120e6_66d4_4643_bd63_428f163eaf8c.slice/crio-55340122a61f5898491913e8e16901625ca0820b63527df020e53105e7d55b88 WatchSource:0}: Error finding container 55340122a61f5898491913e8e16901625ca0820b63527df020e53105e7d55b88: Status 404 returned error can't find the container with id 55340122a61f5898491913e8e16901625ca0820b63527df020e53105e7d55b88 Oct 30 13:16:45.762529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.762488 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:16:45.766602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.766291 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.766602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.766336 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.766602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.766354 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.766602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.766367 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.766602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.766380 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.766602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.766393 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.766602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.766408 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b70acc-a40d-4065-8c58-53a6fff7d969-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.766602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.766422 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm5jz\" (UniqueName: \"kubernetes.io/projected/54b70acc-a40d-4065-8c58-53a6fff7d969-kube-api-access-dm5jz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.971862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.971768 5049 generic.go:356] "Generic (PLEG): container finished" podID="d630a793-11e7-4767-9b82-aa7725d12d14" containerID="e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6" exitCode=0 Oct 30 13:16:45.972316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.971972 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" event={"ID":"d630a793-11e7-4767-9b82-aa7725d12d14","Type":"ContainerDied","Data":"e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6"} Oct 30 13:16:45.972316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.972038 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" event={"ID":"d630a793-11e7-4767-9b82-aa7725d12d14","Type":"ContainerDied","Data":"749a93b9bafa16512fc243a45ef7bec9551a23e4fdcab26c711eb33313d20b46"} Oct 30 13:16:45.972316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.972070 5049 scope.go:122] "RemoveContainer" containerID="e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6" Oct 30 13:16:45.972316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.972262 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4" Oct 30 13:16:45.975890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.975858 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" event={"ID":"20a120e6-66d4-4643-bd63-428f163eaf8c","Type":"ContainerStarted","Data":"55340122a61f5898491913e8e16901625ca0820b63527df020e53105e7d55b88"} Oct 30 13:16:45.977967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.977941 5049 generic.go:356] "Generic (PLEG): container finished" podID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerID="888d95f44fbb9e2a8dd23ffaa58fa512f8f2ef512cf35845e0053160bef0b223" exitCode=0 Oct 30 13:16:45.978092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.978014 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" event={"ID":"6bd729fe-55e6-46d3-bc51-ddac03b27463","Type":"ContainerDied","Data":"888d95f44fbb9e2a8dd23ffaa58fa512f8f2ef512cf35845e0053160bef0b223"} Oct 30 13:16:45.979728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.979706 5049 generic.go:356] "Generic (PLEG): container finished" podID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerID="4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88" exitCode=0 Oct 30 13:16:45.979832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.979759 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" event={"ID":"54b70acc-a40d-4065-8c58-53a6fff7d969","Type":"ContainerDied","Data":"4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88"} Oct 30 13:16:45.979832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.979769 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" Oct 30 13:16:45.979832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:45.979795 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr" event={"ID":"54b70acc-a40d-4065-8c58-53a6fff7d969","Type":"ContainerDied","Data":"673bbe0b98019732a3c4a3a6d4f99e00d1e0a1ad9469397dc5d32e9346f39592"} Oct 30 13:16:46.082206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.082181 5049 scope.go:122] "RemoveContainer" containerID="e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6" Oct 30 13:16:46.082746 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:16:46.082711 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6\": container with ID starting with e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6 not found: ID does not exist" containerID="e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6" Oct 30 13:16:46.082861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.082756 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6"} err="failed to get container status \"e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6\": rpc error: code = NotFound desc = could not find container \"e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6\": container with ID starting with e6620e3688c8971307d96eda41584fe256ccfa4b7c180ffd6485a005d78049f6 not found: ID does not exist" Oct 30 13:16:46.082861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.082779 5049 scope.go:122] "RemoveContainer" containerID="4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88" Oct 30 13:16:46.087189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.087170 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:16:46.097295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.097281 5049 scope.go:122] "RemoveContainer" containerID="4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88" Oct 30 13:16:46.097619 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:16:46.097601 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88\": container with ID starting with 4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88 not found: ID does not exist" containerID="4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88" Oct 30 13:16:46.097663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.097649 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88"} err="failed to get container status \"4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88\": rpc error: code = NotFound desc = could not find container \"4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88\": container with ID starting with 4fea7d3950d620699ad2f13065b04767b95c3580ff5551edc762221265e99c88 not found: ID does not exist" Oct 30 13:16:46.219944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.216183 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4"] Oct 30 13:16:46.219944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.216243 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659d72-6b4fb4ccbd-nr9x4"] Oct 30 13:16:46.234112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.233410 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr"] Oct 30 13:16:46.234112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.233468 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-wc7dr"] Oct 30 13:16:46.281552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.281520 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-kube-api-access-f7jwh\" (UniqueName: \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-kube-api-access-f7jwh\") pod \"6bd729fe-55e6-46d3-bc51-ddac03b27463\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " Oct 30 13:16:46.281680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.281611 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd729fe-55e6-46d3-bc51-ddac03b27463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd729fe-55e6-46d3-bc51-ddac03b27463-istiod-ca-cert\") pod \"6bd729fe-55e6-46d3-bc51-ddac03b27463\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " Oct 30 13:16:46.282027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.282002 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd729fe-55e6-46d3-bc51-ddac03b27463-istiod-ca-cert" pod "6bd729fe-55e6-46d3-bc51-ddac03b27463" (UID: "6bd729fe-55e6-46d3-bc51-ddac03b27463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:46.282237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.282217 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-credential-socket\") pod \"6bd729fe-55e6-46d3-bc51-ddac03b27463\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " Oct 30 13:16:46.282306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.282294 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-envoy\") pod \"6bd729fe-55e6-46d3-bc51-ddac03b27463\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " Oct 30 13:16:46.282544 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.282517 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-credential-socket" pod "6bd729fe-55e6-46d3-bc51-ddac03b27463" (UID: "6bd729fe-55e6-46d3-bc51-ddac03b27463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.282760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.282739 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-socket\") pod \"6bd729fe-55e6-46d3-bc51-ddac03b27463\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " Oct 30 13:16:46.282852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.282835 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-podinfo\") pod \"6bd729fe-55e6-46d3-bc51-ddac03b27463\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " Oct 30 13:16:46.283178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.283025 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-socket" pod "6bd729fe-55e6-46d3-bc51-ddac03b27463" (UID: "6bd729fe-55e6-46d3-bc51-ddac03b27463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.283273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.283181 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-certs\") pod \"6bd729fe-55e6-46d3-bc51-ddac03b27463\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " Oct 30 13:16:46.283273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.283244 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-data\") pod \"6bd729fe-55e6-46d3-bc51-ddac03b27463\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " Oct 30 13:16:46.283513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.283454 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-certs" pod "6bd729fe-55e6-46d3-bc51-ddac03b27463" (UID: "6bd729fe-55e6-46d3-bc51-ddac03b27463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.283592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.283523 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-data" pod "6bd729fe-55e6-46d3-bc51-ddac03b27463" (UID: "6bd729fe-55e6-46d3-bc51-ddac03b27463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.283657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.283640 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-token\" (UniqueName: \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-token\") pod \"6bd729fe-55e6-46d3-bc51-ddac03b27463\" (UID: \"6bd729fe-55e6-46d3-bc51-ddac03b27463\") " Oct 30 13:16:46.284742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.284717 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd729fe-55e6-46d3-bc51-ddac03b27463-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.284838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.284749 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.284838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.284762 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.284838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.284776 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.284838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.284789 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.286896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.286813 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-envoy" pod "6bd729fe-55e6-46d3-bc51-ddac03b27463" (UID: "6bd729fe-55e6-46d3-bc51-ddac03b27463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.286896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.286859 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-kube-api-access-f7jwh" pod "6bd729fe-55e6-46d3-bc51-ddac03b27463" (UID: "6bd729fe-55e6-46d3-bc51-ddac03b27463"). InnerVolumeSpecName "kube-api-access-f7jwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:46.287120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.287090 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-podinfo" pod "6bd729fe-55e6-46d3-bc51-ddac03b27463" (UID: "6bd729fe-55e6-46d3-bc51-ddac03b27463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:46.288939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.288909 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-token" pod "6bd729fe-55e6-46d3-bc51-ddac03b27463" (UID: "6bd729fe-55e6-46d3-bc51-ddac03b27463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:46.388682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.388647 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.388682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.388680 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.388948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.388695 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.388948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.388708 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7jwh\" (UniqueName: \"kubernetes.io/projected/6bd729fe-55e6-46d3-bc51-ddac03b27463-kube-api-access-f7jwh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.396524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.396487 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:16:46.489677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.489586 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-envoy\") pod \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " Oct 30 13:16:46.489837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.489735 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-data\") pod \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " Oct 30 13:16:46.489837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.489781 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-certs\") pod \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " Oct 30 13:16:46.489963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.489868 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-kube-api-access-q2rtk\" (UniqueName: \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-kube-api-access-q2rtk\") pod \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " Oct 30 13:16:46.489963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.489914 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-socket\") pod \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " Oct 30 13:16:46.490084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.489979 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-token\" (UniqueName: \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-token\") pod \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " Oct 30 13:16:46.490084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.490064 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-credential-socket\") pod \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " Oct 30 13:16:46.490198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.490103 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istiod-ca-cert\") pod \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " Oct 30 13:16:46.490198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.490152 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-podinfo\") pod \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\" (UID: \"fc7f05d7-e738-44ca-93ed-f5c02d64084a\") " Oct 30 13:16:46.490449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.490419 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-certs" pod "fc7f05d7-e738-44ca-93ed-f5c02d64084a" (UID: "fc7f05d7-e738-44ca-93ed-f5c02d64084a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.491081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.490860 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istiod-ca-cert" pod "fc7f05d7-e738-44ca-93ed-f5c02d64084a" (UID: "fc7f05d7-e738-44ca-93ed-f5c02d64084a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:46.491081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.490928 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.491353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.491284 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-socket" pod "fc7f05d7-e738-44ca-93ed-f5c02d64084a" (UID: "fc7f05d7-e738-44ca-93ed-f5c02d64084a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.491889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.491865 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-data" pod "fc7f05d7-e738-44ca-93ed-f5c02d64084a" (UID: "fc7f05d7-e738-44ca-93ed-f5c02d64084a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.491987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.491927 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-credential-socket" pod "fc7f05d7-e738-44ca-93ed-f5c02d64084a" (UID: "fc7f05d7-e738-44ca-93ed-f5c02d64084a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.492543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.492515 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-envoy" pod "fc7f05d7-e738-44ca-93ed-f5c02d64084a" (UID: "fc7f05d7-e738-44ca-93ed-f5c02d64084a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.492976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.492950 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-token" pod "fc7f05d7-e738-44ca-93ed-f5c02d64084a" (UID: "fc7f05d7-e738-44ca-93ed-f5c02d64084a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:46.493837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.493810 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-kube-api-access-q2rtk" pod "fc7f05d7-e738-44ca-93ed-f5c02d64084a" (UID: "fc7f05d7-e738-44ca-93ed-f5c02d64084a"). InnerVolumeSpecName "kube-api-access-q2rtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:46.494233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.494210 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-podinfo" pod "fc7f05d7-e738-44ca-93ed-f5c02d64084a" (UID: "fc7f05d7-e738-44ca-93ed-f5c02d64084a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:46.598344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.597979 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.598344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.598022 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.598344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.598038 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.598344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.598051 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.598344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.598064 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.598344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.598077 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2rtk\" (UniqueName: \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-kube-api-access-q2rtk\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.598344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.598090 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7f05d7-e738-44ca-93ed-f5c02d64084a-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.598344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.598103 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7f05d7-e738-44ca-93ed-f5c02d64084a-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.995050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.995016 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" event={"ID":"20a120e6-66d4-4643-bd63-428f163eaf8c","Type":"ContainerStarted","Data":"f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d"} Oct 30 13:16:46.995479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.995083 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:46.996727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.996683 5049 generic.go:356] "Generic (PLEG): container finished" podID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerID="e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571" exitCode=0 Oct 30 13:16:46.996727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.996725 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" Oct 30 13:16:46.996972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.996780 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" event={"ID":"fc7f05d7-e738-44ca-93ed-f5c02d64084a","Type":"ContainerDied","Data":"e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571"} Oct 30 13:16:46.996972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.996875 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz" event={"ID":"fc7f05d7-e738-44ca-93ed-f5c02d64084a","Type":"ContainerDied","Data":"ab4987a82a61996ddf804a99101957123dcd3cc5027baba01cc6669224993519"} Oct 30 13:16:46.996972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.996914 5049 scope.go:122] "RemoveContainer" containerID="e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571" Oct 30 13:16:46.998484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.998466 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" Oct 30 13:16:46.998562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:46.998451 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6" event={"ID":"6bd729fe-55e6-46d3-bc51-ddac03b27463","Type":"ContainerDied","Data":"1d7206d60af51b1701956182d04a22a2fa48347e6a68b0011b4e8fba27872217"} Oct 30 13:16:47.009202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.009186 5049 scope.go:122] "RemoveContainer" containerID="e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571" Oct 30 13:16:47.009469 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:16:47.009453 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571\": container with ID starting with e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571 not found: ID does not exist" containerID="e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571" Oct 30 13:16:47.009551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.009476 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571"} err="failed to get container status \"e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571\": rpc error: code = NotFound desc = could not find container \"e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571\": container with ID starting with e7d076a20576b7facfe05842d396dc5dabcb4154ce6278df21e02d5e1ab0f571 not found: ID does not exist" Oct 30 13:16:47.009551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.009492 5049 scope.go:122] "RemoveContainer" containerID="888d95f44fbb9e2a8dd23ffaa58fa512f8f2ef512cf35845e0053160bef0b223" Oct 30 13:16:47.062335 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.062279 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" podStartSLOduration=2.578634302 podStartE2EDuration="3.062264964s" totalImagesPullingTime="483.630662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:47.058661202 +0000 UTC m=+1513.873855638" watchObservedRunningTime="2026-10-30 13:16:47.062264964 +0000 UTC m=+1513.877459387" Oct 30 13:16:47.117069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.117036 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz"] Oct 30 13:16:47.129549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.129520 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wssbz"] Oct 30 13:16:47.234997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.234921 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6"] Oct 30 13:16:47.273128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.273021 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-tncv6"] Oct 30 13:16:47.343053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.343018 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" path="/var/lib/kubelet/pods/54b70acc-a40d-4065-8c58-53a6fff7d969/volumes" Oct 30 13:16:47.346380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.343779 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" path="/var/lib/kubelet/pods/6bd729fe-55e6-46d3-bc51-ddac03b27463/volumes" Oct 30 13:16:47.346380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.344358 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d630a793-11e7-4767-9b82-aa7725d12d14" path="/var/lib/kubelet/pods/d630a793-11e7-4767-9b82-aa7725d12d14/volumes" Oct 30 13:16:47.346380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.345348 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" path="/var/lib/kubelet/pods/fc7f05d7-e738-44ca-93ed-f5c02d64084a/volumes" Oct 30 13:16:47.757155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:47.757122 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:48.030523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:48.030431 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:48.788284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:48.788250 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:49.076299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:49.076270 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:49.309343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:49.308400 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:50.018846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.018806 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm"] Oct 30 13:16:50.019641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019618 5049 cpu_manager.go:414] "containerMap: removing container" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" Oct 30 13:16:50.019641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019637 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" Oct 30 13:16:50.019803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019664 5049 cpu_manager.go:414] "containerMap: removing container" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerName="istio-proxy" Oct 30 13:16:50.019803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019669 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerName="istio-proxy" Oct 30 13:16:50.019803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019688 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d630a793-11e7-4767-9b82-aa7725d12d14" containerName="payload-processing" Oct 30 13:16:50.019803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019694 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d630a793-11e7-4767-9b82-aa7725d12d14" containerName="payload-processing" Oct 30 13:16:50.019803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019707 5049 cpu_manager.go:414] "containerMap: removing container" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerName="istio-proxy" Oct 30 13:16:50.019803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019711 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerName="istio-proxy" Oct 30 13:16:50.019990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019843 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d630a793-11e7-4767-9b82-aa7725d12d14" containerName="payload-processing" Oct 30 13:16:50.019990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019853 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd729fe-55e6-46d3-bc51-ddac03b27463" containerName="istio-proxy" Oct 30 13:16:50.019990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019864 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7f05d7-e738-44ca-93ed-f5c02d64084a" containerName="istio-proxy" Oct 30 13:16:50.019990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.019875 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b70acc-a40d-4065-8c58-53a6fff7d969" containerName="istio-proxy" Oct 30 13:16:50.025529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.025507 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.062443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.062411 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm"] Oct 30 13:16:50.148352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.143072 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:16:50.148352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.143264 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:16:50.175762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.175612 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:50.209966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.208422 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b0881-82hrm\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.209966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.208994 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrmz\" (UniqueName: \"kubernetes.io/projected/cbad5317-0a2d-468f-9097-d19046b8292c-kube-api-access-jgrmz\") pod \"maas-api-revoke-keys-e2e-api-b-6b0881-82hrm\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.209966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.209134 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b0881-82hrm\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.310900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.310870 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrmz\" (UniqueName: \"kubernetes.io/projected/cbad5317-0a2d-468f-9097-d19046b8292c-kube-api-access-jgrmz\") pod \"maas-api-revoke-keys-e2e-api-b-6b0881-82hrm\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.311218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.310976 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b0881-82hrm\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.311218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.311032 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b0881-82hrm\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.311691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.311663 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b0881-82hrm\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.313628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.313599 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b0881-82hrm\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.346361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.346300 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrmz\" (UniqueName: \"kubernetes.io/projected/cbad5317-0a2d-468f-9097-d19046b8292c-kube-api-access-jgrmz\") pod \"maas-api-revoke-keys-e2e-api-b-6b0881-82hrm\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.640671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.640590 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:50.972984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:50.972959 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm"] Oct 30 13:16:50.976501 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:16:50.976468 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbad5317_0a2d_468f_9097_d19046b8292c.slice/crio-75d0a6cad618db4480284a7c1775c61e20f561cab1eb45fcd987a2fbdc7d2332 WatchSource:0}: Error finding container 75d0a6cad618db4480284a7c1775c61e20f561cab1eb45fcd987a2fbdc7d2332: Status 404 returned error can't find the container with id 75d0a6cad618db4480284a7c1775c61e20f561cab1eb45fcd987a2fbdc7d2332 Oct 30 13:16:51.060386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:51.060346 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" event={"ID":"cbad5317-0a2d-468f-9097-d19046b8292c","Type":"ContainerStarted","Data":"75d0a6cad618db4480284a7c1775c61e20f561cab1eb45fcd987a2fbdc7d2332"} Oct 30 13:16:52.071283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:52.071249 5049 generic.go:356] "Generic (PLEG): container finished" podID="cbad5317-0a2d-468f-9097-d19046b8292c" containerID="4622f88121f1a7df6a25484dd7d939ed0e27bea8ca396032aa331aca040fe652" exitCode=0 Oct 30 13:16:52.071794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:52.071352 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" event={"ID":"cbad5317-0a2d-468f-9097-d19046b8292c","Type":"ContainerDied","Data":"4622f88121f1a7df6a25484dd7d939ed0e27bea8ca396032aa331aca040fe652"} Oct 30 13:16:52.621342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:52.620583 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:52.801801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:52.801770 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:53.018280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.018199 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:53.026986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.026963 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:16:53.298067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.298038 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:53.378040 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.378003 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-xks-ca\") pod \"cbad5317-0a2d-468f-9097-d19046b8292c\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " Oct 30 13:16:53.378227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.378047 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbad5317-0a2d-468f-9097-d19046b8292c-kube-api-access-jgrmz\" (UniqueName: \"kubernetes.io/projected/cbad5317-0a2d-468f-9097-d19046b8292c-kube-api-access-jgrmz\") pod \"cbad5317-0a2d-468f-9097-d19046b8292c\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " Oct 30 13:16:53.378297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.378285 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-openshift-ca\") pod \"cbad5317-0a2d-468f-9097-d19046b8292c\" (UID: \"cbad5317-0a2d-468f-9097-d19046b8292c\") " Oct 30 13:16:53.378808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.378777 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-openshift-ca" pod "cbad5317-0a2d-468f-9097-d19046b8292c" (UID: "cbad5317-0a2d-468f-9097-d19046b8292c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:53.380298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.380276 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-xks-ca" pod "cbad5317-0a2d-468f-9097-d19046b8292c" (UID: "cbad5317-0a2d-468f-9097-d19046b8292c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:16:53.380595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.380578 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbad5317-0a2d-468f-9097-d19046b8292c-kube-api-access-jgrmz" pod "cbad5317-0a2d-468f-9097-d19046b8292c" (UID: "cbad5317-0a2d-468f-9097-d19046b8292c"). InnerVolumeSpecName "kube-api-access-jgrmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:53.480242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.480205 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:53.480242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.480237 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbad5317-0a2d-468f-9097-d19046b8292c-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:53.480242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:53.480250 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgrmz\" (UniqueName: \"kubernetes.io/projected/cbad5317-0a2d-468f-9097-d19046b8292c-kube-api-access-jgrmz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.098820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.098772 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" event={"ID":"cbad5317-0a2d-468f-9097-d19046b8292c","Type":"ContainerDied","Data":"75d0a6cad618db4480284a7c1775c61e20f561cab1eb45fcd987a2fbdc7d2332"} Oct 30 13:16:54.099083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.099065 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75d0a6cad618db4480284a7c1775c61e20f561cab1eb45fcd987a2fbdc7d2332" Oct 30 13:16:54.099248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.099236 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm" Oct 30 13:16:54.526952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.526864 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s"] Oct 30 13:16:54.527382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.527118 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerName="istio-proxy" containerID="cri-o://71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4" gracePeriod=30 Oct 30 13:16:54.534310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.533911 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl"] Oct 30 13:16:54.534310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.534154 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerName="istio-proxy" containerID="cri-o://c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f" gracePeriod=30 Oct 30 13:16:54.540246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.540220 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh"] Oct 30 13:16:54.540678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.540648 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" containerID="cri-o://4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630" gracePeriod=30 Oct 30 13:16:54.546456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.546431 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2"] Oct 30 13:16:54.546644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:54.546622 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerName="istio-proxy" containerID="cri-o://91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7" gracePeriod=30 Oct 30 13:16:55.161348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:55.159029 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:56.678747 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:16:56.678710 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:16:57.350684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:57.350648 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:59.401361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:59.401310 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:59.696350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:59.694933 5049 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:16:59.696350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:59.694985 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 13:16:59.896538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:59.896511 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:16:59.905288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:16:59.905231 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:17:00.000845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.000813 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:17:00.005387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.005359 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:17:00.038217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038180 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-podinfo\") pod \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " Oct 30 13:17:00.038427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038245 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istiod-ca-cert\") pod \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " Oct 30 13:17:00.038427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038279 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-certs\") pod \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " Oct 30 13:17:00.038427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038343 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-kube-api-access-mdj9g\" (UniqueName: \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-kube-api-access-mdj9g\") pod \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " Oct 30 13:17:00.038427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038385 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-socket\") pod \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " Oct 30 13:17:00.038673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038498 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-data\") pod \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " Oct 30 13:17:00.038673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038615 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-credential-socket\") pod \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " Oct 30 13:17:00.038776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038679 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-kube-api-access-dk24m\" (UniqueName: \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-kube-api-access-dk24m\") pod \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " Oct 30 13:17:00.038776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038710 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-token\" (UniqueName: \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-token\") pod \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " Oct 30 13:17:00.038776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038756 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-podinfo\") pod \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " Oct 30 13:17:00.038927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038793 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-certs\") pod \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " Oct 30 13:17:00.038927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038845 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-envoy\") pod \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " Oct 30 13:17:00.038927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038898 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-socket\") pod \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " Oct 30 13:17:00.039079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.038958 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istiod-ca-cert\") pod \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " Oct 30 13:17:00.039079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.039042 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-token\" (UniqueName: \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-token\") pod \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " Oct 30 13:17:00.039182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.039096 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-credential-socket\") pod \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\" (UID: \"951a9e8c-d9b8-47a8-a02a-56c94cea5bba\") " Oct 30 13:17:00.039182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.039129 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-envoy\") pod \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " Oct 30 13:17:00.039295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.039229 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-data\") pod \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\" (UID: \"86bd54d1-91ab-4e96-a4cc-2ad4726018ba\") " Oct 30 13:17:00.042706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.040634 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-data" pod "86bd54d1-91ab-4e96-a4cc-2ad4726018ba" (UID: "86bd54d1-91ab-4e96-a4cc-2ad4726018ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.042706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.041130 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-data" pod "951a9e8c-d9b8-47a8-a02a-56c94cea5bba" (UID: "951a9e8c-d9b8-47a8-a02a-56c94cea5bba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.042706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.041355 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-socket" pod "86bd54d1-91ab-4e96-a4cc-2ad4726018ba" (UID: "86bd54d1-91ab-4e96-a4cc-2ad4726018ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.042706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.041489 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-socket" pod "951a9e8c-d9b8-47a8-a02a-56c94cea5bba" (UID: "951a9e8c-d9b8-47a8-a02a-56c94cea5bba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.042706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.041594 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-credential-socket" pod "86bd54d1-91ab-4e96-a4cc-2ad4726018ba" (UID: "86bd54d1-91ab-4e96-a4cc-2ad4726018ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.042706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.041792 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istiod-ca-cert" pod "86bd54d1-91ab-4e96-a4cc-2ad4726018ba" (UID: "86bd54d1-91ab-4e96-a4cc-2ad4726018ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:00.042706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.042007 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-certs" pod "86bd54d1-91ab-4e96-a4cc-2ad4726018ba" (UID: "86bd54d1-91ab-4e96-a4cc-2ad4726018ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.042706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.042099 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istiod-ca-cert" pod "951a9e8c-d9b8-47a8-a02a-56c94cea5bba" (UID: "951a9e8c-d9b8-47a8-a02a-56c94cea5bba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:00.042706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.042301 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-certs" pod "951a9e8c-d9b8-47a8-a02a-56c94cea5bba" (UID: "951a9e8c-d9b8-47a8-a02a-56c94cea5bba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.042706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.042462 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-credential-socket" pod "951a9e8c-d9b8-47a8-a02a-56c94cea5bba" (UID: "951a9e8c-d9b8-47a8-a02a-56c94cea5bba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.058879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.057759 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-envoy" pod "86bd54d1-91ab-4e96-a4cc-2ad4726018ba" (UID: "86bd54d1-91ab-4e96-a4cc-2ad4726018ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.058879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.057826 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-kube-api-access-dk24m" pod "951a9e8c-d9b8-47a8-a02a-56c94cea5bba" (UID: "951a9e8c-d9b8-47a8-a02a-56c94cea5bba"). InnerVolumeSpecName "kube-api-access-dk24m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:00.059072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.058953 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-podinfo" pod "951a9e8c-d9b8-47a8-a02a-56c94cea5bba" (UID: "951a9e8c-d9b8-47a8-a02a-56c94cea5bba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:17:00.059126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.059106 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-kube-api-access-mdj9g" pod "86bd54d1-91ab-4e96-a4cc-2ad4726018ba" (UID: "86bd54d1-91ab-4e96-a4cc-2ad4726018ba"). InnerVolumeSpecName "kube-api-access-mdj9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:00.059518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.059495 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-envoy" pod "951a9e8c-d9b8-47a8-a02a-56c94cea5bba" (UID: "951a9e8c-d9b8-47a8-a02a-56c94cea5bba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.059635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.059500 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-token" pod "86bd54d1-91ab-4e96-a4cc-2ad4726018ba" (UID: "86bd54d1-91ab-4e96-a4cc-2ad4726018ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:00.059700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.059652 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-podinfo" pod "86bd54d1-91ab-4e96-a4cc-2ad4726018ba" (UID: "86bd54d1-91ab-4e96-a4cc-2ad4726018ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:17:00.060205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.060174 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-token" pod "951a9e8c-d9b8-47a8-a02a-56c94cea5bba" (UID: "951a9e8c-d9b8-47a8-a02a-56c94cea5bba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:00.140582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.140549 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-kube-api-access-4znbg\" (UniqueName: \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-kube-api-access-4znbg\") pod \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " Oct 30 13:17:00.140788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.140598 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-token\" (UniqueName: \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-token\") pod \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " Oct 30 13:17:00.140788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.140690 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-certs\") pod \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " Oct 30 13:17:00.140788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.140767 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-data\") pod \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " Oct 30 13:17:00.140956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.140841 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-podinfo\") pod \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " Oct 30 13:17:00.140956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.140874 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istiod-ca-cert\") pod \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " Oct 30 13:17:00.140956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.140927 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-socket\") pod \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " Oct 30 13:17:00.141108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.140993 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-token\" (UniqueName: \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-token\") pod \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " Oct 30 13:17:00.141108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141023 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-certs" pod "1f0bd8bb-ae7a-40ad-950c-058013605dd6" (UID: "1f0bd8bb-ae7a-40ad-950c-058013605dd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.141874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141372 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istiod-ca-cert" pod "ae7d2b91-83e5-48e0-9c32-7f2228d2c532" (UID: "ae7d2b91-83e5-48e0-9c32-7f2228d2c532"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:00.141874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141422 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-credential-socket\") pod \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " Oct 30 13:17:00.141874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141538 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-podinfo\") pod \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " Oct 30 13:17:00.141874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141577 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istiod-ca-cert\") pod \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " Oct 30 13:17:00.141874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141628 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-socket\") pod \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " Oct 30 13:17:00.141874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141674 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-socket" pod "1f0bd8bb-ae7a-40ad-950c-058013605dd6" (UID: "1f0bd8bb-ae7a-40ad-950c-058013605dd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.141874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141731 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-credential-socket" pod "ae7d2b91-83e5-48e0-9c32-7f2228d2c532" (UID: "ae7d2b91-83e5-48e0-9c32-7f2228d2c532"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.141874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141739 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-data\") pod \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " Oct 30 13:17:00.141874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141799 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-kube-api-access-wkmt4\" (UniqueName: \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-kube-api-access-wkmt4\") pod \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " Oct 30 13:17:00.141874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141879 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-envoy\") pod \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " Oct 30 13:17:00.143184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141930 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-credential-socket\") pod \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\" (UID: \"1f0bd8bb-ae7a-40ad-950c-058013605dd6\") " Oct 30 13:17:00.143184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141966 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-certs\") pod \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " Oct 30 13:17:00.143184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.141993 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istiod-ca-cert" pod "1f0bd8bb-ae7a-40ad-950c-058013605dd6" (UID: "1f0bd8bb-ae7a-40ad-950c-058013605dd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:00.143184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.142288 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-data" pod "1f0bd8bb-ae7a-40ad-950c-058013605dd6" (UID: "1f0bd8bb-ae7a-40ad-950c-058013605dd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.143184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.142569 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-data" pod "ae7d2b91-83e5-48e0-9c32-7f2228d2c532" (UID: "ae7d2b91-83e5-48e0-9c32-7f2228d2c532"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.143184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.142575 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-socket" pod "ae7d2b91-83e5-48e0-9c32-7f2228d2c532" (UID: "ae7d2b91-83e5-48e0-9c32-7f2228d2c532"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.143683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.143107 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-envoy\") pod \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\" (UID: \"ae7d2b91-83e5-48e0-9c32-7f2228d2c532\") " Oct 30 13:17:00.143775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.143349 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-certs" pod "ae7d2b91-83e5-48e0-9c32-7f2228d2c532" (UID: "ae7d2b91-83e5-48e0-9c32-7f2228d2c532"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.143775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.143361 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-credential-socket" pod "1f0bd8bb-ae7a-40ad-950c-058013605dd6" (UID: "1f0bd8bb-ae7a-40ad-950c-058013605dd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.144249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144201 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-kube-api-access-4znbg" pod "1f0bd8bb-ae7a-40ad-950c-058013605dd6" (UID: "1f0bd8bb-ae7a-40ad-950c-058013605dd6"). InnerVolumeSpecName "kube-api-access-4znbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:00.144389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144275 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144298 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144312 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144341 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144355 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144368 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144380 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144392 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144405 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144419 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144432 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144446 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdj9g\" (UniqueName: \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-kube-api-access-mdj9g\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144460 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144473 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144486 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144499 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144511 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144524 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144536 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144548 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144564 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144577 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk24m\" (UniqueName: \"kubernetes.io/projected/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-kube-api-access-dk24m\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144590 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bd54d1-91ab-4e96-a4cc-2ad4726018ba-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144602 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144615 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144627 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144640 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144654 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951a9e8c-d9b8-47a8-a02a-56c94cea5bba-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.144751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.144636 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-token" pod "ae7d2b91-83e5-48e0-9c32-7f2228d2c532" (UID: "ae7d2b91-83e5-48e0-9c32-7f2228d2c532"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:00.146602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.146561 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-envoy" pod "1f0bd8bb-ae7a-40ad-950c-058013605dd6" (UID: "1f0bd8bb-ae7a-40ad-950c-058013605dd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.146602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.146585 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-podinfo" pod "ae7d2b91-83e5-48e0-9c32-7f2228d2c532" (UID: "ae7d2b91-83e5-48e0-9c32-7f2228d2c532"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:17:00.146851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.146825 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-podinfo" pod "1f0bd8bb-ae7a-40ad-950c-058013605dd6" (UID: "1f0bd8bb-ae7a-40ad-950c-058013605dd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:17:00.147403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.147379 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-kube-api-access-wkmt4" pod "ae7d2b91-83e5-48e0-9c32-7f2228d2c532" (UID: "ae7d2b91-83e5-48e0-9c32-7f2228d2c532"). InnerVolumeSpecName "kube-api-access-wkmt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:00.148462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.148426 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-envoy" pod "ae7d2b91-83e5-48e0-9c32-7f2228d2c532" (UID: "ae7d2b91-83e5-48e0-9c32-7f2228d2c532"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:00.149063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.149036 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-token" pod "1f0bd8bb-ae7a-40ad-950c-058013605dd6" (UID: "1f0bd8bb-ae7a-40ad-950c-058013605dd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:00.167073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.166985 5049 generic.go:356] "Generic (PLEG): container finished" podID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerID="c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f" exitCode=0 Oct 30 13:17:00.167244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.167098 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" event={"ID":"ae7d2b91-83e5-48e0-9c32-7f2228d2c532","Type":"ContainerDied","Data":"c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f"} Oct 30 13:17:00.167244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.167146 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" event={"ID":"ae7d2b91-83e5-48e0-9c32-7f2228d2c532","Type":"ContainerDied","Data":"c79e15c20cb5581fa1ef7f234d8e408aa62bca23033cee72519798d949841585"} Oct 30 13:17:00.167244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.167175 5049 scope.go:122] "RemoveContainer" containerID="c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f" Oct 30 13:17:00.167502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.167364 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl" Oct 30 13:17:00.173889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.173841 5049 generic.go:356] "Generic (PLEG): container finished" podID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerID="4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630" exitCode=0 Oct 30 13:17:00.174339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.174251 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" event={"ID":"951a9e8c-d9b8-47a8-a02a-56c94cea5bba","Type":"ContainerDied","Data":"4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630"} Oct 30 13:17:00.174597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.174503 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" event={"ID":"951a9e8c-d9b8-47a8-a02a-56c94cea5bba","Type":"ContainerDied","Data":"46a73b8a28244106b89f12cfde248610b7cda13771840f1cb1aa2770b40e9778"} Oct 30 13:17:00.174882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.174852 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh" Oct 30 13:17:00.186897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.180620 5049 generic.go:356] "Generic (PLEG): container finished" podID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerID="71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4" exitCode=0 Oct 30 13:17:00.186897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.180722 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" Oct 30 13:17:00.186897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.180721 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" event={"ID":"86bd54d1-91ab-4e96-a4cc-2ad4726018ba","Type":"ContainerDied","Data":"71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4"} Oct 30 13:17:00.186897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.181198 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s" event={"ID":"86bd54d1-91ab-4e96-a4cc-2ad4726018ba","Type":"ContainerDied","Data":"fee01d9f6e0ee0f10bb3f3a9e6e57a0f9d9d34ad8f64d4a497013680c9ed70f2"} Oct 30 13:17:00.192789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.191739 5049 scope.go:122] "RemoveContainer" containerID="c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f" Oct 30 13:17:00.192789 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:17:00.192042 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f\": container with ID starting with c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f not found: ID does not exist" containerID="c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f" Oct 30 13:17:00.192789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.192080 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f"} err="failed to get container status \"c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f\": rpc error: code = NotFound desc = could not find container \"c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f\": container with ID starting with c4cafde088d074040566a29fc060afee0ddb2c214fda3e53a24d4e2523f6fb1f not found: ID does not exist" Oct 30 13:17:00.192789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.192102 5049 scope.go:122] "RemoveContainer" containerID="4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630" Oct 30 13:17:00.192789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.192224 5049 generic.go:356] "Generic (PLEG): container finished" podID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerID="91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7" exitCode=0 Oct 30 13:17:00.192789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.192308 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" Oct 30 13:17:00.192789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.192419 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" event={"ID":"1f0bd8bb-ae7a-40ad-950c-058013605dd6","Type":"ContainerDied","Data":"91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7"} Oct 30 13:17:00.192789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.192455 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2" event={"ID":"1f0bd8bb-ae7a-40ad-950c-058013605dd6","Type":"ContainerDied","Data":"1fc63f16b79f3aebeb156d53edf55454b1051078fa009195c8938db840e347fc"} Oct 30 13:17:00.207704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.207678 5049 scope.go:122] "RemoveContainer" containerID="4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630" Oct 30 13:17:00.208078 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:17:00.208036 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630\": container with ID starting with 4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630 not found: ID does not exist" containerID="4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630" Oct 30 13:17:00.208166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.208089 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630"} err="failed to get container status \"4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630\": rpc error: code = NotFound desc = could not find container \"4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630\": container with ID starting with 4b060f02f67b4984b4d1c66d45a25454ed0ba4d9d58878ff418d45f14937e630 not found: ID does not exist" Oct 30 13:17:00.208166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.208112 5049 scope.go:122] "RemoveContainer" containerID="71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4" Oct 30 13:17:00.225760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.225084 5049 scope.go:122] "RemoveContainer" containerID="71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4" Oct 30 13:17:00.226196 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:17:00.226169 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4\": container with ID starting with 71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4 not found: ID does not exist" containerID="71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4" Oct 30 13:17:00.226283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.226208 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4"} err="failed to get container status \"71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4\": rpc error: code = NotFound desc = could not find container \"71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4\": container with ID starting with 71a473b090e2d3762c270c30fbc0c3c4e1b0a7d555d570b448049f7e2ddb9fb4 not found: ID does not exist" Oct 30 13:17:00.226283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.226230 5049 scope.go:122] "RemoveContainer" containerID="91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7" Oct 30 13:17:00.230381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.230352 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl"] Oct 30 13:17:00.240098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.240070 5049 scope.go:122] "RemoveContainer" containerID="91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7" Oct 30 13:17:00.240539 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:17:00.240512 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7\": container with ID starting with 91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7 not found: ID does not exist" containerID="91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7" Oct 30 13:17:00.240647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.240550 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7"} err="failed to get container status \"91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7\": rpc error: code = NotFound desc = could not find container \"91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7\": container with ID starting with 91884ff257c4c0f9c6b0373cb0ad786f09fec61e632381b2ace7735e8bc343d7 not found: ID does not exist" Oct 30 13:17:00.242859 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.242839 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-mfzrl"] Oct 30 13:17:00.245751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.245734 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.245805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.245754 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.245805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.245764 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.245805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.245773 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkmt4\" (UniqueName: \"kubernetes.io/projected/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-kube-api-access-wkmt4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.245805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.245783 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.245805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.245793 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7d2b91-83e5-48e0-9c32-7f2228d2c532-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.245805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.245801 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4znbg\" (UniqueName: \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-kube-api-access-4znbg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.245986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.245811 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0bd8bb-ae7a-40ad-950c-058013605dd6-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:00.305942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.305910 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s"] Oct 30 13:17:00.316442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.316410 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-5p94s"] Oct 30 13:17:00.364454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.364418 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh"] Oct 30 13:17:00.371631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.371600 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-dnhfh"] Oct 30 13:17:00.415453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.415419 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2"] Oct 30 13:17:00.420032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.419955 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-kdct2"] Oct 30 13:17:00.895756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:00.895702 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:01.060880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:01.060834 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:01.388871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:01.388836 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" path="/var/lib/kubelet/pods/1f0bd8bb-ae7a-40ad-950c-058013605dd6/volumes" Oct 30 13:17:01.399430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:01.390847 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" path="/var/lib/kubelet/pods/86bd54d1-91ab-4e96-a4cc-2ad4726018ba/volumes" Oct 30 13:17:01.401678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:01.401648 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" path="/var/lib/kubelet/pods/951a9e8c-d9b8-47a8-a02a-56c94cea5bba/volumes" Oct 30 13:17:01.402843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:01.402817 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" path="/var/lib/kubelet/pods/ae7d2b91-83e5-48e0-9c32-7f2228d2c532/volumes" Oct 30 13:17:01.660344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:01.660235 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:03.892551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:03.892517 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:06.102367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:06.102241 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:08.474824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:08.474545 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:10.584379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:10.584253 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:12.645301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:12.645269 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:14.809495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:14.809457 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:16.616812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:16.616067 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:17.294192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.294163 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs"] Oct 30 13:17:17.294922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.294907 5049 cpu_manager.go:414] "containerMap: removing container" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerName="istio-proxy" Oct 30 13:17:17.294966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.294923 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerName="istio-proxy" Oct 30 13:17:17.294966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.294937 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerName="istio-proxy" Oct 30 13:17:17.294966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.294942 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerName="istio-proxy" Oct 30 13:17:17.294966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.294958 5049 cpu_manager.go:414] "containerMap: removing container" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerName="istio-proxy" Oct 30 13:17:17.294966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.294963 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerName="istio-proxy" Oct 30 13:17:17.295130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.294987 5049 cpu_manager.go:414] "containerMap: removing container" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" Oct 30 13:17:17.295130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.294992 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" Oct 30 13:17:17.295130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.295016 5049 cpu_manager.go:414] "containerMap: removing container" podUID="cbad5317-0a2d-468f-9097-d19046b8292c" containerName="revoke-keys" Oct 30 13:17:17.295130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.295021 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbad5317-0a2d-468f-9097-d19046b8292c" containerName="revoke-keys" Oct 30 13:17:17.295130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.295124 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0bd8bb-ae7a-40ad-950c-058013605dd6" containerName="istio-proxy" Oct 30 13:17:17.295299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.295133 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7d2b91-83e5-48e0-9c32-7f2228d2c532" containerName="istio-proxy" Oct 30 13:17:17.295299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.295140 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="86bd54d1-91ab-4e96-a4cc-2ad4726018ba" containerName="istio-proxy" Oct 30 13:17:17.295299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.295152 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="951a9e8c-d9b8-47a8-a02a-56c94cea5bba" containerName="istio-proxy" Oct 30 13:17:17.295299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.295162 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbad5317-0a2d-468f-9097-d19046b8292c" containerName="revoke-keys" Oct 30 13:17:17.300816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.300794 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.315806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.315780 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:17:17.335109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.335077 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:17:17.346503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.346474 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs"] Oct 30 13:17:17.385907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.385858 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5989e3ed-224rs\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.386097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.386075 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5989e3ed-224rs\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.386250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.386236 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzcc2\" (UniqueName: \"kubernetes.io/projected/e8c126fe-c2ba-43dd-b3da-f420b25146d8-kube-api-access-rzcc2\") pod \"maas-api-revoke-sub-5989e3ed-224rs\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.488124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.488086 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5989e3ed-224rs\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.488351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.488216 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5989e3ed-224rs\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.488445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.488365 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzcc2\" (UniqueName: \"kubernetes.io/projected/e8c126fe-c2ba-43dd-b3da-f420b25146d8-kube-api-access-rzcc2\") pod \"maas-api-revoke-sub-5989e3ed-224rs\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.488918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.488897 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5989e3ed-224rs\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.490492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.490473 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5989e3ed-224rs\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.510128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.510099 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzcc2\" (UniqueName: \"kubernetes.io/projected/e8c126fe-c2ba-43dd-b3da-f420b25146d8-kube-api-access-rzcc2\") pod \"maas-api-revoke-sub-5989e3ed-224rs\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.618448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.618359 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:17.935150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:17.935117 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs"] Oct 30 13:17:17.937087 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:17:17.937048 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c126fe_c2ba_43dd_b3da_f420b25146d8.slice/crio-fa80ce36758c993282156e86eb18fa96e1cdd45a1b3625668ad093f8e87e9890 WatchSource:0}: Error finding container fa80ce36758c993282156e86eb18fa96e1cdd45a1b3625668ad093f8e87e9890: Status 404 returned error can't find the container with id fa80ce36758c993282156e86eb18fa96e1cdd45a1b3625668ad093f8e87e9890 Oct 30 13:17:18.540782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:18.540685 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" event={"ID":"e8c126fe-c2ba-43dd-b3da-f420b25146d8","Type":"ContainerStarted","Data":"6374648881ec58c043871a8afd0af3200acf04f5c6272cec9f5a0ced65992cd3"} Oct 30 13:17:18.540782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:18.540751 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" event={"ID":"e8c126fe-c2ba-43dd-b3da-f420b25146d8","Type":"ContainerStarted","Data":"fa80ce36758c993282156e86eb18fa96e1cdd45a1b3625668ad093f8e87e9890"} Oct 30 13:17:18.615854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:18.615781 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" podStartSLOduration=1.6157584360000001 podStartE2EDuration="1.615758436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:18.604660844 +0000 UTC m=+1545.419855266" watchObservedRunningTime="2026-10-30 13:17:18.615758436 +0000 UTC m=+1545.430952860" Oct 30 13:17:18.757906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:18.757870 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:19.552654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:19.552623 5049 generic.go:356] "Generic (PLEG): container finished" podID="e8c126fe-c2ba-43dd-b3da-f420b25146d8" containerID="6374648881ec58c043871a8afd0af3200acf04f5c6272cec9f5a0ced65992cd3" exitCode=0 Oct 30 13:17:19.552831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:19.552684 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" event={"ID":"e8c126fe-c2ba-43dd-b3da-f420b25146d8","Type":"ContainerDied","Data":"6374648881ec58c043871a8afd0af3200acf04f5c6272cec9f5a0ced65992cd3"} Oct 30 13:17:20.719974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.719933 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:20.753922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.753889 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:20.857288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.856007 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-xks-ca\") pod \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " Oct 30 13:17:20.857288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.856092 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c126fe-c2ba-43dd-b3da-f420b25146d8-kube-api-access-rzcc2\" (UniqueName: \"kubernetes.io/projected/e8c126fe-c2ba-43dd-b3da-f420b25146d8-kube-api-access-rzcc2\") pod \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " Oct 30 13:17:20.857288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.856475 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-openshift-ca\") pod \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\" (UID: \"e8c126fe-c2ba-43dd-b3da-f420b25146d8\") " Oct 30 13:17:20.857288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.856930 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-openshift-ca" pod "e8c126fe-c2ba-43dd-b3da-f420b25146d8" (UID: "e8c126fe-c2ba-43dd-b3da-f420b25146d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:20.857663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.857364 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:20.859643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.859611 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-xks-ca" pod "e8c126fe-c2ba-43dd-b3da-f420b25146d8" (UID: "e8c126fe-c2ba-43dd-b3da-f420b25146d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:20.862345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.862064 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c126fe-c2ba-43dd-b3da-f420b25146d8-kube-api-access-rzcc2" pod "e8c126fe-c2ba-43dd-b3da-f420b25146d8" (UID: "e8c126fe-c2ba-43dd-b3da-f420b25146d8"). InnerVolumeSpecName "kube-api-access-rzcc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:20.909797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.909757 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:20.959671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.959641 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c126fe-c2ba-43dd-b3da-f420b25146d8-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:20.959671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:20.959669 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzcc2\" (UniqueName: \"kubernetes.io/projected/e8c126fe-c2ba-43dd-b3da-f420b25146d8-kube-api-access-rzcc2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:21.572211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:21.572185 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" Oct 30 13:17:21.572401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:21.572230 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs" event={"ID":"e8c126fe-c2ba-43dd-b3da-f420b25146d8","Type":"ContainerDied","Data":"fa80ce36758c993282156e86eb18fa96e1cdd45a1b3625668ad093f8e87e9890"} Oct 30 13:17:21.572401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:21.572278 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa80ce36758c993282156e86eb18fa96e1cdd45a1b3625668ad093f8e87e9890" Oct 30 13:17:22.740358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.739896 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:22.821862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.821833 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck"] Oct 30 13:17:22.822702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.822679 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e8c126fe-c2ba-43dd-b3da-f420b25146d8" containerName="revoke-keys" Oct 30 13:17:22.822702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.822701 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c126fe-c2ba-43dd-b3da-f420b25146d8" containerName="revoke-keys" Oct 30 13:17:22.822947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.822932 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c126fe-c2ba-43dd-b3da-f420b25146d8" containerName="revoke-keys" Oct 30 13:17:22.826992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.826969 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:22.838370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.837659 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck"] Oct 30 13:17:22.843344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.842013 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:17:22.843344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.842112 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:17:22.989828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.989797 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fcb69-scjck\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:22.990004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.989868 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fcb69-scjck\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:22.990004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:22.989940 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dd45\" (UniqueName: \"kubernetes.io/projected/610d502a-0f69-482a-87fb-c82ed83c026d-kube-api-access-2dd45\") pod \"maas-api-revoke-keys-e2e-api-a-9fcb69-scjck\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:23.091132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:23.091085 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fcb69-scjck\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:23.091385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:23.091175 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fcb69-scjck\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:23.091385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:23.091203 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dd45\" (UniqueName: \"kubernetes.io/projected/610d502a-0f69-482a-87fb-c82ed83c026d-kube-api-access-2dd45\") pod \"maas-api-revoke-keys-e2e-api-a-9fcb69-scjck\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:23.092016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:23.091986 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fcb69-scjck\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:23.093669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:23.093647 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fcb69-scjck\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:23.113537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:23.113504 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dd45\" (UniqueName: \"kubernetes.io/projected/610d502a-0f69-482a-87fb-c82ed83c026d-kube-api-access-2dd45\") pod \"maas-api-revoke-keys-e2e-api-a-9fcb69-scjck\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:23.147886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:23.147856 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:23.464181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:23.464155 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck"] Oct 30 13:17:23.467691 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:17:23.467661 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610d502a_0f69_482a_87fb_c82ed83c026d.slice/crio-b297d03a4177911fa9c2bcec45398f7ee65fd90ce6cd564040b274b125977b06 WatchSource:0}: Error finding container b297d03a4177911fa9c2bcec45398f7ee65fd90ce6cd564040b274b125977b06: Status 404 returned error can't find the container with id b297d03a4177911fa9c2bcec45398f7ee65fd90ce6cd564040b274b125977b06 Oct 30 13:17:23.589703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:23.589668 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" event={"ID":"610d502a-0f69-482a-87fb-c82ed83c026d","Type":"ContainerStarted","Data":"b297d03a4177911fa9c2bcec45398f7ee65fd90ce6cd564040b274b125977b06"} Oct 30 13:17:24.599855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:24.599821 5049 generic.go:356] "Generic (PLEG): container finished" podID="610d502a-0f69-482a-87fb-c82ed83c026d" containerID="5ce4bdc404135658e724dab5f8b5c3d28a89fca7e9288aa0ea387c727115affe" exitCode=0 Oct 30 13:17:24.600313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:24.599898 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" event={"ID":"610d502a-0f69-482a-87fb-c82ed83c026d","Type":"ContainerDied","Data":"5ce4bdc404135658e724dab5f8b5c3d28a89fca7e9288aa0ea387c727115affe"} Oct 30 13:17:24.972274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:24.971257 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:25.819406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:25.819378 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:25.935021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:25.934934 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610d502a-0f69-482a-87fb-c82ed83c026d-kube-api-access-2dd45\" (UniqueName: \"kubernetes.io/projected/610d502a-0f69-482a-87fb-c82ed83c026d-kube-api-access-2dd45\") pod \"610d502a-0f69-482a-87fb-c82ed83c026d\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " Oct 30 13:17:25.935213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:25.935193 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-xks-ca\") pod \"610d502a-0f69-482a-87fb-c82ed83c026d\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " Oct 30 13:17:25.935377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:25.935359 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-openshift-ca\") pod \"610d502a-0f69-482a-87fb-c82ed83c026d\" (UID: \"610d502a-0f69-482a-87fb-c82ed83c026d\") " Oct 30 13:17:25.935781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:25.935750 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-openshift-ca" pod "610d502a-0f69-482a-87fb-c82ed83c026d" (UID: "610d502a-0f69-482a-87fb-c82ed83c026d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:25.936246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:25.936224 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:25.937193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:25.937169 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610d502a-0f69-482a-87fb-c82ed83c026d-kube-api-access-2dd45" pod "610d502a-0f69-482a-87fb-c82ed83c026d" (UID: "610d502a-0f69-482a-87fb-c82ed83c026d"). InnerVolumeSpecName "kube-api-access-2dd45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:25.937355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:25.937305 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-xks-ca" pod "610d502a-0f69-482a-87fb-c82ed83c026d" (UID: "610d502a-0f69-482a-87fb-c82ed83c026d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:26.037992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:26.037963 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610d502a-0f69-482a-87fb-c82ed83c026d-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:26.037992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:26.037992 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dd45\" (UniqueName: \"kubernetes.io/projected/610d502a-0f69-482a-87fb-c82ed83c026d-kube-api-access-2dd45\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:26.619601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:26.619566 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" event={"ID":"610d502a-0f69-482a-87fb-c82ed83c026d","Type":"ContainerDied","Data":"b297d03a4177911fa9c2bcec45398f7ee65fd90ce6cd564040b274b125977b06"} Oct 30 13:17:26.619799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:26.619626 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b297d03a4177911fa9c2bcec45398f7ee65fd90ce6cd564040b274b125977b06" Oct 30 13:17:26.619799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:26.619576 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck" Oct 30 13:17:26.679028 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:17:26.678993 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:17:26.944382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:26.944281 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:27.097582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:27.097547 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:27.658238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:27.658207 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:29.081182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:29.081150 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:31.215729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:31.215693 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:33.360563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:33.360530 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:33.715961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:33.715876 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v"] Oct 30 13:17:33.866580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:33.866547 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:34.158862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:34.158827 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:34.850420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:34.850383 5049 scope.go:122] "RemoveContainer" containerID="0483cb4ddbeb6ddcf1a0e7521c6136e8422c06519e14c062ff794774cfdfe615" Oct 30 13:17:34.861502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:34.861480 5049 scope.go:122] "RemoveContainer" containerID="9188c6cfa4de381431932c93162065cef4dda9167970942aca0f18e2975939aa" Oct 30 13:17:34.871132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:34.871114 5049 scope.go:122] "RemoveContainer" containerID="9946a0cab3d8c12a0f43ef72cb64f780f7cc06d11d8040e0228fcf9bb87c94b8" Oct 30 13:17:34.880397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:34.880379 5049 scope.go:122] "RemoveContainer" containerID="90610a1926b3e5530d577ff99b7ee9b20ac702e8eac1904f540d24619d580113" Oct 30 13:17:35.214179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.214078 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:35.651373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.647341 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm"] Oct 30 13:17:35.651373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.648551 5049 cpu_manager.go:414] "containerMap: removing container" podUID="610d502a-0f69-482a-87fb-c82ed83c026d" containerName="revoke-keys" Oct 30 13:17:35.651373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.648572 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610d502a-0f69-482a-87fb-c82ed83c026d" containerName="revoke-keys" Oct 30 13:17:35.651373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.648810 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="610d502a-0f69-482a-87fb-c82ed83c026d" containerName="revoke-keys" Oct 30 13:17:35.657091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.657058 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.671921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.670426 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-77254d-openshift-default-dockercfg-4wm72\"" Oct 30 13:17:35.687852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.687804 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm"] Oct 30 13:17:35.756694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.756656 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:35.792163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.792131 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istiod-ca-cert\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.792396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.792193 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-token\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.792396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.792284 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-data\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.792396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.792388 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-podinfo\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.792574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.792441 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw4nh\" (UniqueName: \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-kube-api-access-jw4nh\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.792630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.792571 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.792689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.792635 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-credential-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.792689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.792680 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-envoy\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.792795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.792733 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-certs\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.901938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.895305 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-certs\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.901938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.895442 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istiod-ca-cert\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.901938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.895513 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-token\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.901938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.895559 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-data\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.901938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.895642 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-podinfo\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.901938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.895702 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw4nh\" (UniqueName: \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-kube-api-access-jw4nh\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.901938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.895795 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.901938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.895868 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-credential-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.901938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.895927 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-envoy\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.908048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.904752 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-data\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.908048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.905019 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-certs\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.908048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.905701 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istiod-ca-cert\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.908048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.907224 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.908048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.907451 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-credential-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.944346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.943970 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-podinfo\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.971392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.966285 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-envoy\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.995382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.995088 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-token\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:35.997542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:35.996156 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw4nh\" (UniqueName: \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-kube-api-access-jw4nh\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:36.076980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:36.076937 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:36.280651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:36.280559 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:36.618917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:36.618889 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm"] Oct 30 13:17:36.622787 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:17:36.622747 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eacaf2d_bdf9_4f8c_918e_44f1910a7467.slice/crio-5b3e43fc6180a7dbb4031dc492b20d1e15069d9752a642c0c842b6d63295eb60 WatchSource:0}: Error finding container 5b3e43fc6180a7dbb4031dc492b20d1e15069d9752a642c0c842b6d63295eb60: Status 404 returned error can't find the container with id 5b3e43fc6180a7dbb4031dc492b20d1e15069d9752a642c0c842b6d63295eb60 Oct 30 13:17:36.626001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:36.625971 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:17:36.626105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:36.626037 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:17:36.626105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:36.626065 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:17:36.712763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:36.712729 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" event={"ID":"6eacaf2d-bdf9-4f8c-918e-44f1910a7467","Type":"ContainerStarted","Data":"93780e70c673bbe26c2f511141359f5e8064a8b310a8eb705ef9a38c34a6129f"} Oct 30 13:17:36.712917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:36.712790 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" event={"ID":"6eacaf2d-bdf9-4f8c-918e-44f1910a7467","Type":"ContainerStarted","Data":"5b3e43fc6180a7dbb4031dc492b20d1e15069d9752a642c0c842b6d63295eb60"} Oct 30 13:17:36.758493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:36.758433 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" podStartSLOduration=1.758415366 podStartE2EDuration="1.758415366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:36.755146755 +0000 UTC m=+1563.570341180" watchObservedRunningTime="2026-10-30 13:17:36.758415366 +0000 UTC m=+1563.573609790" Oct 30 13:17:37.144990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:37.144938 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:37.281108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:37.281073 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:37.287447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:37.287422 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:37.722339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:37.722294 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:37.723773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:37.723748 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:17:38.035373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:38.035209 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:38.717210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:38.717133 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" podUID="af82fe26-4c25-478d-b576-8d7740721735" containerName="payload-pre-processing" containerID="cri-o://cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50" gracePeriod=25 Oct 30 13:17:38.814238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:38.814199 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:39.064702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.064675 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:17:39.184150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.184114 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af82fe26-4c25-478d-b576-8d7740721735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af82fe26-4c25-478d-b576-8d7740721735-plugins-config-volume\") pod \"af82fe26-4c25-478d-b576-8d7740721735\" (UID: \"af82fe26-4c25-478d-b576-8d7740721735\") " Oct 30 13:17:39.184355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.184241 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af82fe26-4c25-478d-b576-8d7740721735-kube-api-access-6vfqj\" (UniqueName: \"kubernetes.io/projected/af82fe26-4c25-478d-b576-8d7740721735-kube-api-access-6vfqj\") pod \"af82fe26-4c25-478d-b576-8d7740721735\" (UID: \"af82fe26-4c25-478d-b576-8d7740721735\") " Oct 30 13:17:39.184530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.184504 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af82fe26-4c25-478d-b576-8d7740721735-plugins-config-volume" pod "af82fe26-4c25-478d-b576-8d7740721735" (UID: "af82fe26-4c25-478d-b576-8d7740721735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:39.184921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.184894 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af82fe26-4c25-478d-b576-8d7740721735-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:39.186367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.186341 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af82fe26-4c25-478d-b576-8d7740721735-kube-api-access-6vfqj" pod "af82fe26-4c25-478d-b576-8d7740721735" (UID: "af82fe26-4c25-478d-b576-8d7740721735"). InnerVolumeSpecName "kube-api-access-6vfqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:39.286605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.286531 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vfqj\" (UniqueName: \"kubernetes.io/projected/af82fe26-4c25-478d-b576-8d7740721735-kube-api-access-6vfqj\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:39.300518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.300481 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:39.478114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.478077 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:39.615033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.615001 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr"] Oct 30 13:17:39.615870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.615847 5049 cpu_manager.go:414] "containerMap: removing container" podUID="af82fe26-4c25-478d-b576-8d7740721735" containerName="payload-pre-processing" Oct 30 13:17:39.615870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.615869 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af82fe26-4c25-478d-b576-8d7740721735" containerName="payload-pre-processing" Oct 30 13:17:39.616108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.616093 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="af82fe26-4c25-478d-b576-8d7740721735" containerName="payload-pre-processing" Oct 30 13:17:39.620586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.620561 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:39.651883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.651850 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:17:39.656834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.656808 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:17:39.663615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.663586 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr"] Oct 30 13:17:39.692740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.692709 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:39.692928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.692824 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:39.693245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.693214 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmtt\" (UniqueName: \"kubernetes.io/projected/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-kube-api-access-thmtt\") pod \"maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:39.742649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.742613 5049 generic.go:356] "Generic (PLEG): container finished" podID="af82fe26-4c25-478d-b576-8d7740721735" containerID="cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50" exitCode=0 Oct 30 13:17:39.743082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.742660 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" Oct 30 13:17:39.743082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.742707 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" event={"ID":"af82fe26-4c25-478d-b576-8d7740721735","Type":"ContainerDied","Data":"cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50"} Oct 30 13:17:39.743082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.742762 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v" event={"ID":"af82fe26-4c25-478d-b576-8d7740721735","Type":"ContainerDied","Data":"b1d7cdbb899ecf72ef0fa33fd8b72b3c49eabccb9d4624d14b463b645c46846b"} Oct 30 13:17:39.743082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.742794 5049 scope.go:122] "RemoveContainer" containerID="cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50" Oct 30 13:17:39.756397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.756374 5049 scope.go:122] "RemoveContainer" containerID="cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50" Oct 30 13:17:39.756706 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:17:39.756685 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50\": container with ID starting with cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50 not found: ID does not exist" containerID="cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50" Oct 30 13:17:39.756760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.756715 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50"} err="failed to get container status \"cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50\": rpc error: code = NotFound desc = could not find container \"cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50\": container with ID starting with cf8bba3377e3a92a000c62d4aa120ffd3356fdeafdff1ac407fcc38aa34f5c50 not found: ID does not exist" Oct 30 13:17:39.792926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.792897 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v"] Oct 30 13:17:39.794852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.794827 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmtt\" (UniqueName: \"kubernetes.io/projected/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-kube-api-access-thmtt\") pod \"maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:39.794940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.794874 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:39.795005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.794941 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:39.795667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.795640 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:39.797676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.797649 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:39.802294 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.802271 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9fcb69-5bb6cf7c55-d8s9v"] Oct 30 13:17:39.817105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.817073 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmtt\" (UniqueName: \"kubernetes.io/projected/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-kube-api-access-thmtt\") pod \"maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:39.938930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:39.938841 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:40.296043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:40.296005 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr"] Oct 30 13:17:40.299876 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:17:40.299821 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8e7ae2_7c14_46c4_a348_d371a9a0466e.slice/crio-63ef9228dd4a01d546090d1ab247c9c801b8d7f4046061962c86ee0817771f92 WatchSource:0}: Error finding container 63ef9228dd4a01d546090d1ab247c9c801b8d7f4046061962c86ee0817771f92: Status 404 returned error can't find the container with id 63ef9228dd4a01d546090d1ab247c9c801b8d7f4046061962c86ee0817771f92 Oct 30 13:17:40.357436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:40.357393 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:40.453311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:40.453279 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9"] Oct 30 13:17:40.453523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:40.453499 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" containerID="cri-o://591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98" gracePeriod=30 Oct 30 13:17:40.757033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:40.756996 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" event={"ID":"3f8e7ae2-7c14-46c4-a348-d371a9a0466e","Type":"ContainerStarted","Data":"63ef9228dd4a01d546090d1ab247c9c801b8d7f4046061962c86ee0817771f92"} Oct 30 13:17:41.342766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.342726 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af82fe26-4c25-478d-b576-8d7740721735" path="/var/lib/kubelet/pods/af82fe26-4c25-478d-b576-8d7740721735/volumes" Oct 30 13:17:41.515562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.515338 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:41.593191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.593119 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx"] Oct 30 13:17:41.597716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.597699 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:17:41.617789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.617765 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-77254d\"" Oct 30 13:17:41.624283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.624246 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx"] Oct 30 13:17:41.645814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.645781 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx"] Oct 30 13:17:41.651831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.651806 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:17:41.671446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.670735 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-77254d-dockercfg-dzplg\"" Oct 30 13:17:41.673639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.673611 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx"] Oct 30 13:17:41.698211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.698179 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:41.808353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.806073 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcpc\" (UniqueName: \"kubernetes.io/projected/d622eb69-bf74-4c55-89fe-18d963542243-kube-api-access-jkcpc\") pod \"payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx\" (UID: \"d622eb69-bf74-4c55-89fe-18d963542243\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:17:41.808353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.806224 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640a2237-2051-4ba3-8aac-5c617e742bb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx\" (UID: \"640a2237-2051-4ba3-8aac-5c617e742bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:17:41.808353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.806276 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glt86\" (UniqueName: \"kubernetes.io/projected/640a2237-2051-4ba3-8aac-5c617e742bb3-kube-api-access-glt86\") pod \"payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx\" (UID: \"640a2237-2051-4ba3-8aac-5c617e742bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:17:41.808353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.806358 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d622eb69-bf74-4c55-89fe-18d963542243-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx\" (UID: \"d622eb69-bf74-4c55-89fe-18d963542243\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:17:41.972448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.968599 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcpc\" (UniqueName: \"kubernetes.io/projected/d622eb69-bf74-4c55-89fe-18d963542243-kube-api-access-jkcpc\") pod \"payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx\" (UID: \"d622eb69-bf74-4c55-89fe-18d963542243\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:17:41.972448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.968850 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640a2237-2051-4ba3-8aac-5c617e742bb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx\" (UID: \"640a2237-2051-4ba3-8aac-5c617e742bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:17:41.972448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.968924 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glt86\" (UniqueName: \"kubernetes.io/projected/640a2237-2051-4ba3-8aac-5c617e742bb3-kube-api-access-glt86\") pod \"payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx\" (UID: \"640a2237-2051-4ba3-8aac-5c617e742bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:17:41.972448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.968988 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d622eb69-bf74-4c55-89fe-18d963542243-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx\" (UID: \"d622eb69-bf74-4c55-89fe-18d963542243\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:17:41.972448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.971438 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640a2237-2051-4ba3-8aac-5c617e742bb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx\" (UID: \"640a2237-2051-4ba3-8aac-5c617e742bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:17:41.972448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.971784 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d622eb69-bf74-4c55-89fe-18d963542243-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx\" (UID: \"d622eb69-bf74-4c55-89fe-18d963542243\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:17:41.981737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.981703 5049 generic.go:356] "Generic (PLEG): container finished" podID="3f8e7ae2-7c14-46c4-a348-d371a9a0466e" containerID="3aa02ddc7e0bf6e2aba6d151cf50da5bca28121ed313c9458c15c0440fcdccae" exitCode=0 Oct 30 13:17:41.982552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.982030 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" event={"ID":"3f8e7ae2-7c14-46c4-a348-d371a9a0466e","Type":"ContainerDied","Data":"3aa02ddc7e0bf6e2aba6d151cf50da5bca28121ed313c9458c15c0440fcdccae"} Oct 30 13:17:41.984843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.984821 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcpc\" (UniqueName: \"kubernetes.io/projected/d622eb69-bf74-4c55-89fe-18d963542243-kube-api-access-jkcpc\") pod \"payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx\" (UID: \"d622eb69-bf74-4c55-89fe-18d963542243\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:17:41.993529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:41.993507 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glt86\" (UniqueName: \"kubernetes.io/projected/640a2237-2051-4ba3-8aac-5c617e742bb3-kube-api-access-glt86\") pod \"payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx\" (UID: \"640a2237-2051-4ba3-8aac-5c617e742bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:17:42.212584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.212541 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:17:42.267687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.267657 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:17:42.532237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.532203 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx"] Oct 30 13:17:42.535100 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:17:42.535068 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd622eb69_bf74_4c55_89fe_18d963542243.slice/crio-161b74d53d5518cd8d824434b50f2dd3f133837124631ebc4dd73a2283087d96 WatchSource:0}: Error finding container 161b74d53d5518cd8d824434b50f2dd3f133837124631ebc4dd73a2283087d96: Status 404 returned error can't find the container with id 161b74d53d5518cd8d824434b50f2dd3f133837124631ebc4dd73a2283087d96 Oct 30 13:17:42.598399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.597050 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:42.652268 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:17:42.652226 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640a2237_2051_4ba3_8aac_5c617e742bb3.slice/crio-df1d99df050ce7bc2eb4c16063b974da80632156b87645c6524ef27fbe7cd9f7 WatchSource:0}: Error finding container df1d99df050ce7bc2eb4c16063b974da80632156b87645c6524ef27fbe7cd9f7: Status 404 returned error can't find the container with id df1d99df050ce7bc2eb4c16063b974da80632156b87645c6524ef27fbe7cd9f7 Oct 30 13:17:42.668096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.668064 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx"] Oct 30 13:17:42.992533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.992497 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" event={"ID":"640a2237-2051-4ba3-8aac-5c617e742bb3","Type":"ContainerStarted","Data":"98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77"} Oct 30 13:17:42.992920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.992603 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:17:42.992920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.992624 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" event={"ID":"640a2237-2051-4ba3-8aac-5c617e742bb3","Type":"ContainerStarted","Data":"df1d99df050ce7bc2eb4c16063b974da80632156b87645c6524ef27fbe7cd9f7"} Oct 30 13:17:42.994596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.994565 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" event={"ID":"d622eb69-bf74-4c55-89fe-18d963542243","Type":"ContainerStarted","Data":"fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7"} Oct 30 13:17:42.994746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.994608 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" event={"ID":"d622eb69-bf74-4c55-89fe-18d963542243","Type":"ContainerStarted","Data":"161b74d53d5518cd8d824434b50f2dd3f133837124631ebc4dd73a2283087d96"} Oct 30 13:17:42.994746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:42.994656 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:17:43.043345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.042267 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" podStartSLOduration=2.042248151 podStartE2EDuration="2.042248151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:43.0337645 +0000 UTC m=+1569.848958925" watchObservedRunningTime="2026-10-30 13:17:43.042248151 +0000 UTC m=+1569.857442576" Oct 30 13:17:43.071404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.071279 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" podStartSLOduration=2.071260259 podStartE2EDuration="2.071260259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:43.070544721 +0000 UTC m=+1569.885739144" watchObservedRunningTime="2026-10-30 13:17:43.071260259 +0000 UTC m=+1569.886454680" Oct 30 13:17:43.261595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.261560 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:43.343666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.343584 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-openshift-ca\") pod \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " Oct 30 13:17:43.343834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.343703 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-kube-api-access-thmtt\" (UniqueName: \"kubernetes.io/projected/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-kube-api-access-thmtt\") pod \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " Oct 30 13:17:43.343834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.343801 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-xks-ca\") pod \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\" (UID: \"3f8e7ae2-7c14-46c4-a348-d371a9a0466e\") " Oct 30 13:17:43.345022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.344996 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-openshift-ca" pod "3f8e7ae2-7c14-46c4-a348-d371a9a0466e" (UID: "3f8e7ae2-7c14-46c4-a348-d371a9a0466e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:43.354599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.354501 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-xks-ca" pod "3f8e7ae2-7c14-46c4-a348-d371a9a0466e" (UID: "3f8e7ae2-7c14-46c4-a348-d371a9a0466e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:43.354960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.354841 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-kube-api-access-thmtt" pod "3f8e7ae2-7c14-46c4-a348-d371a9a0466e" (UID: "3f8e7ae2-7c14-46c4-a348-d371a9a0466e"). InnerVolumeSpecName "kube-api-access-thmtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:43.382433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.382400 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt"] Oct 30 13:17:43.383335 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.383303 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3f8e7ae2-7c14-46c4-a348-d371a9a0466e" containerName="revoke-keys" Oct 30 13:17:43.383402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.383337 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8e7ae2-7c14-46c4-a348-d371a9a0466e" containerName="revoke-keys" Oct 30 13:17:43.383528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.383513 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8e7ae2-7c14-46c4-a348-d371a9a0466e" containerName="revoke-keys" Oct 30 13:17:43.387602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.387583 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:43.408522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.408489 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt"] Oct 30 13:17:43.446137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.446105 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:43.446344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.446239 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:43.446431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.446396 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2fdt\" (UniqueName: \"kubernetes.io/projected/c884ad38-4f81-45fa-a61d-43ae56e63732-kube-api-access-l2fdt\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:43.446648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.446567 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:43.446648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.446611 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:43.446648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.446630 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thmtt\" (UniqueName: \"kubernetes.io/projected/3f8e7ae2-7c14-46c4-a348-d371a9a0466e-kube-api-access-thmtt\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:43.548204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.548168 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fdt\" (UniqueName: \"kubernetes.io/projected/c884ad38-4f81-45fa-a61d-43ae56e63732-kube-api-access-l2fdt\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:43.548427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.548349 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:43.548516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.548499 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:43.549393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.549367 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:43.551145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.551120 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:43.569753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.569723 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fdt\" (UniqueName: \"kubernetes.io/projected/c884ad38-4f81-45fa-a61d-43ae56e63732-kube-api-access-l2fdt\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:43.705515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:43.705433 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:44.004962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:44.004880 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" Oct 30 13:17:44.004962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:44.004905 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr" event={"ID":"3f8e7ae2-7c14-46c4-a348-d371a9a0466e","Type":"ContainerDied","Data":"63ef9228dd4a01d546090d1ab247c9c801b8d7f4046061962c86ee0817771f92"} Oct 30 13:17:44.005446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:44.004968 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63ef9228dd4a01d546090d1ab247c9c801b8d7f4046061962c86ee0817771f92" Oct 30 13:17:44.019195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:44.019162 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt"] Oct 30 13:17:44.019463 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:17:44.019438 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc884ad38_4f81_45fa_a61d_43ae56e63732.slice/crio-74248f7627fa9acd97ed30ae71111be9f6b8e99b74d6bcce959249d76a77525c WatchSource:0}: Error finding container 74248f7627fa9acd97ed30ae71111be9f6b8e99b74d6bcce959249d76a77525c: Status 404 returned error can't find the container with id 74248f7627fa9acd97ed30ae71111be9f6b8e99b74d6bcce959249d76a77525c Oct 30 13:17:44.587300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:44.584009 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:45.021139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:45.021103 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" event={"ID":"c884ad38-4f81-45fa-a61d-43ae56e63732","Type":"ContainerStarted","Data":"6006244c9b03bfe5874023d03b3f1698bbd703bd9e140c8e64d18f840b54e1e8"} Oct 30 13:17:45.021537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:45.021167 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" event={"ID":"c884ad38-4f81-45fa-a61d-43ae56e63732","Type":"ContainerStarted","Data":"74248f7627fa9acd97ed30ae71111be9f6b8e99b74d6bcce959249d76a77525c"} Oct 30 13:17:45.030183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:45.030158 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj"] Oct 30 13:17:45.039495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:45.039467 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b39a67e-6swfj"] Oct 30 13:17:45.354606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:45.354572 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b" path="/var/lib/kubelet/pods/2a758ab5-9ffe-4fce-8b99-5fbdf0d6c30b/volumes" Oct 30 13:17:45.929202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:45.929172 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:17:46.031076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.031033 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-credential-socket\") pod \"cbb76201-354c-4b8c-8f6b-2483135657c6\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " Oct 30 13:17:46.031605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.031221 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-envoy\") pod \"cbb76201-354c-4b8c-8f6b-2483135657c6\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " Oct 30 13:17:46.031605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.031283 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-credential-socket" pod "cbb76201-354c-4b8c-8f6b-2483135657c6" (UID: "cbb76201-354c-4b8c-8f6b-2483135657c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:46.031605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.031364 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-token\" (UniqueName: \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-token\") pod \"cbb76201-354c-4b8c-8f6b-2483135657c6\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " Oct 30 13:17:46.031605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.031429 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-certs\") pod \"cbb76201-354c-4b8c-8f6b-2483135657c6\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " Oct 30 13:17:46.031605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.031491 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-podinfo\") pod \"cbb76201-354c-4b8c-8f6b-2483135657c6\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " Oct 30 13:17:46.031881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.031721 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb76201-354c-4b8c-8f6b-2483135657c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb76201-354c-4b8c-8f6b-2483135657c6-istiod-ca-cert\") pod \"cbb76201-354c-4b8c-8f6b-2483135657c6\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " Oct 30 13:17:46.031881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.031755 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-kube-api-access-88987\" (UniqueName: \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-kube-api-access-88987\") pod \"cbb76201-354c-4b8c-8f6b-2483135657c6\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " Oct 30 13:17:46.031881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.031811 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-socket\") pod \"cbb76201-354c-4b8c-8f6b-2483135657c6\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " Oct 30 13:17:46.032042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.031874 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-data\") pod \"cbb76201-354c-4b8c-8f6b-2483135657c6\" (UID: \"cbb76201-354c-4b8c-8f6b-2483135657c6\") " Oct 30 13:17:46.032716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.032597 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb76201-354c-4b8c-8f6b-2483135657c6-istiod-ca-cert" pod "cbb76201-354c-4b8c-8f6b-2483135657c6" (UID: "cbb76201-354c-4b8c-8f6b-2483135657c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:46.032845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.032773 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb76201-354c-4b8c-8f6b-2483135657c6-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.032845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.032798 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.032967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.032839 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-certs" pod "cbb76201-354c-4b8c-8f6b-2483135657c6" (UID: "cbb76201-354c-4b8c-8f6b-2483135657c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:46.034073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.034044 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-envoy" pod "cbb76201-354c-4b8c-8f6b-2483135657c6" (UID: "cbb76201-354c-4b8c-8f6b-2483135657c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:46.034073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.034056 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-token" pod "cbb76201-354c-4b8c-8f6b-2483135657c6" (UID: "cbb76201-354c-4b8c-8f6b-2483135657c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:46.034353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.034312 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-socket" pod "cbb76201-354c-4b8c-8f6b-2483135657c6" (UID: "cbb76201-354c-4b8c-8f6b-2483135657c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:46.034562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.034538 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-data" pod "cbb76201-354c-4b8c-8f6b-2483135657c6" (UID: "cbb76201-354c-4b8c-8f6b-2483135657c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:17:46.034757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.034735 5049 generic.go:356] "Generic (PLEG): container finished" podID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerID="591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98" exitCode=0 Oct 30 13:17:46.034940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.034922 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" Oct 30 13:17:46.035376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.035339 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" event={"ID":"cbb76201-354c-4b8c-8f6b-2483135657c6","Type":"ContainerDied","Data":"591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98"} Oct 30 13:17:46.035479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.035406 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9" event={"ID":"cbb76201-354c-4b8c-8f6b-2483135657c6","Type":"ContainerDied","Data":"616d70b2db36b459607a5ab0c598427cd7a5a75904e833c6faf098b48cb258ee"} Oct 30 13:17:46.035479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.035443 5049 scope.go:122] "RemoveContainer" containerID="591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98" Oct 30 13:17:46.035848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.035695 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-kube-api-access-88987" pod "cbb76201-354c-4b8c-8f6b-2483135657c6" (UID: "cbb76201-354c-4b8c-8f6b-2483135657c6"). InnerVolumeSpecName "kube-api-access-88987". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:46.035848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.035750 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-podinfo" pod "cbb76201-354c-4b8c-8f6b-2483135657c6" (UID: "cbb76201-354c-4b8c-8f6b-2483135657c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:17:46.044849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.044821 5049 generic.go:356] "Generic (PLEG): container finished" podID="c884ad38-4f81-45fa-a61d-43ae56e63732" containerID="6006244c9b03bfe5874023d03b3f1698bbd703bd9e140c8e64d18f840b54e1e8" exitCode=0 Oct 30 13:17:46.044960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.044895 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" event={"ID":"c884ad38-4f81-45fa-a61d-43ae56e63732","Type":"ContainerDied","Data":"6006244c9b03bfe5874023d03b3f1698bbd703bd9e140c8e64d18f840b54e1e8"} Oct 30 13:17:46.101626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.100864 5049 scope.go:122] "RemoveContainer" containerID="591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98" Oct 30 13:17:46.102162 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:17:46.102044 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98\": container with ID starting with 591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98 not found: ID does not exist" containerID="591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98" Oct 30 13:17:46.102162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.102094 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98"} err="failed to get container status \"591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98\": rpc error: code = NotFound desc = could not find container \"591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98\": container with ID starting with 591aa9f4715adf8aff3755e3fb06d2120a6272739c1cac34a680126a400c6e98 not found: ID does not exist" Oct 30 13:17:46.134245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.134206 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88987\" (UniqueName: \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-kube-api-access-88987\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.134245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.134247 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.134487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.134260 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.134487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.134271 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.134487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.134282 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.134487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.134292 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb76201-354c-4b8c-8f6b-2483135657c6-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.134487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.134303 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb76201-354c-4b8c-8f6b-2483135657c6-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.273251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.273220 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:46.340916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.340880 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c884ad38-4f81-45fa-a61d-43ae56e63732-kube-api-access-l2fdt\" (UniqueName: \"kubernetes.io/projected/c884ad38-4f81-45fa-a61d-43ae56e63732-kube-api-access-l2fdt\") pod \"c884ad38-4f81-45fa-a61d-43ae56e63732\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " Oct 30 13:17:46.341097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.341043 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-openshift-ca\") pod \"c884ad38-4f81-45fa-a61d-43ae56e63732\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " Oct 30 13:17:46.341222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.341201 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-xks-ca\") pod \"c884ad38-4f81-45fa-a61d-43ae56e63732\" (UID: \"c884ad38-4f81-45fa-a61d-43ae56e63732\") " Oct 30 13:17:46.342990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.342952 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-openshift-ca" pod "c884ad38-4f81-45fa-a61d-43ae56e63732" (UID: "c884ad38-4f81-45fa-a61d-43ae56e63732"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:46.344456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.344424 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-xks-ca" pod "c884ad38-4f81-45fa-a61d-43ae56e63732" (UID: "c884ad38-4f81-45fa-a61d-43ae56e63732"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:46.345552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.345527 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c884ad38-4f81-45fa-a61d-43ae56e63732-kube-api-access-l2fdt" pod "c884ad38-4f81-45fa-a61d-43ae56e63732" (UID: "c884ad38-4f81-45fa-a61d-43ae56e63732"). InnerVolumeSpecName "kube-api-access-l2fdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:46.407906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.407865 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9"] Oct 30 13:17:46.416779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.416747 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-xkgv9"] Oct 30 13:17:46.443488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.443448 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.443488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.443485 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2fdt\" (UniqueName: \"kubernetes.io/projected/c884ad38-4f81-45fa-a61d-43ae56e63732-kube-api-access-l2fdt\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.443691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.443499 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c884ad38-4f81-45fa-a61d-43ae56e63732-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:46.696876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:46.696843 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:47.068369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:47.066112 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" Oct 30 13:17:47.068369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:47.066896 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt" event={"ID":"c884ad38-4f81-45fa-a61d-43ae56e63732","Type":"ContainerDied","Data":"74248f7627fa9acd97ed30ae71111be9f6b8e99b74d6bcce959249d76a77525c"} Oct 30 13:17:47.068369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:47.066949 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74248f7627fa9acd97ed30ae71111be9f6b8e99b74d6bcce959249d76a77525c" Oct 30 13:17:47.419388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:47.418632 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" path="/var/lib/kubelet/pods/cbb76201-354c-4b8c-8f6b-2483135657c6/volumes" Oct 30 13:17:49.167144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:49.167092 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:49.333242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:49.333194 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 13:17:49.338435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:49.334508 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:49.402542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:49.402498 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:50.277015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:50.276984 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:50.557049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:50.557019 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:50.827703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:50.827611 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl"] Oct 30 13:17:50.827884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:50.827815 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" podUID="58aba26d-82d5-4d0b-8d39-b4109b823511" containerName="maas-api" containerID="cri-o://7c8fe8e39d3ac4994330335adcfaa3732e7496c36b1bf89746befa917e5be1a9" gracePeriod=30 Oct 30 13:17:51.001015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.000977 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:51.489738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.489702 5049 generic.go:356] "Generic (PLEG): container finished" podID="58aba26d-82d5-4d0b-8d39-b4109b823511" containerID="7c8fe8e39d3ac4994330335adcfaa3732e7496c36b1bf89746befa917e5be1a9" exitCode=0 Oct 30 13:17:51.495190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.495160 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" event={"ID":"58aba26d-82d5-4d0b-8d39-b4109b823511","Type":"ContainerDied","Data":"7c8fe8e39d3ac4994330335adcfaa3732e7496c36b1bf89746befa917e5be1a9"} Oct 30 13:17:51.500797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.500769 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:17:51.575765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.575697 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58aba26d-82d5-4d0b-8d39-b4109b823511-kube-api-access-f8lx2\" (UniqueName: \"kubernetes.io/projected/58aba26d-82d5-4d0b-8d39-b4109b823511-kube-api-access-f8lx2\") pod \"58aba26d-82d5-4d0b-8d39-b4109b823511\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " Oct 30 13:17:51.576086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.576058 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-maas-api-tls\") pod \"58aba26d-82d5-4d0b-8d39-b4109b823511\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " Oct 30 13:17:51.576236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.576218 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58aba26d-82d5-4d0b-8d39-b4109b823511-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58aba26d-82d5-4d0b-8d39-b4109b823511-openshift-service-ca\") pod \"58aba26d-82d5-4d0b-8d39-b4109b823511\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " Oct 30 13:17:51.576413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.576397 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-metrics-certs\") pod \"58aba26d-82d5-4d0b-8d39-b4109b823511\" (UID: \"58aba26d-82d5-4d0b-8d39-b4109b823511\") " Oct 30 13:17:51.578462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.577255 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58aba26d-82d5-4d0b-8d39-b4109b823511-openshift-service-ca" pod "58aba26d-82d5-4d0b-8d39-b4109b823511" (UID: "58aba26d-82d5-4d0b-8d39-b4109b823511"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:51.581099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.581077 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-metrics-certs" pod "58aba26d-82d5-4d0b-8d39-b4109b823511" (UID: "58aba26d-82d5-4d0b-8d39-b4109b823511"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:51.581214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.581120 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-maas-api-tls" pod "58aba26d-82d5-4d0b-8d39-b4109b823511" (UID: "58aba26d-82d5-4d0b-8d39-b4109b823511"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:51.582839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.582813 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aba26d-82d5-4d0b-8d39-b4109b823511-kube-api-access-f8lx2" pod "58aba26d-82d5-4d0b-8d39-b4109b823511" (UID: "58aba26d-82d5-4d0b-8d39-b4109b823511"). InnerVolumeSpecName "kube-api-access-f8lx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:51.679667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.679580 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8lx2\" (UniqueName: \"kubernetes.io/projected/58aba26d-82d5-4d0b-8d39-b4109b823511-kube-api-access-f8lx2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:51.679667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.679620 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:51.679667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.679633 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58aba26d-82d5-4d0b-8d39-b4109b823511-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:51.679667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.679646 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58aba26d-82d5-4d0b-8d39-b4109b823511-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:17:51.713468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.713434 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:51.878253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:51.878219 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:52.364496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:52.364463 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:52.502885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:52.502839 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" Oct 30 13:17:52.502885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:52.502872 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl" event={"ID":"58aba26d-82d5-4d0b-8d39-b4109b823511","Type":"ContainerDied","Data":"5ddd04974b2dd0eab5d4ce3350d44383aead7409b09c31fa299249a8d3a025b4"} Oct 30 13:17:52.503461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:52.502942 5049 scope.go:122] "RemoveContainer" containerID="7c8fe8e39d3ac4994330335adcfaa3732e7496c36b1bf89746befa917e5be1a9" Oct 30 13:17:52.602744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:52.602712 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl"] Oct 30 13:17:52.615906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:52.615825 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659d72-77bcc947b5-lrvgl"] Oct 30 13:17:53.351059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:53.351021 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58aba26d-82d5-4d0b-8d39-b4109b823511" path="/var/lib/kubelet/pods/58aba26d-82d5-4d0b-8d39-b4109b823511/volumes" Oct 30 13:17:53.923203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:53.923173 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:54.008511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:54.008483 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:17:54.008677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:54.008560 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:17:54.110280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:54.110243 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:55.914350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:55.914178 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:56.679522 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:17:56.679487 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:17:57.785273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:57.785184 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:59.032618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.032569 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:59.197553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.197517 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:59.722664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.722616 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:59.934373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.934334 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:59.995027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.994931 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9"] Oct 30 13:17:59.995950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.995920 5049 cpu_manager.go:414] "containerMap: removing container" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" Oct 30 13:17:59.995950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.995949 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" Oct 30 13:17:59.996155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.995995 5049 cpu_manager.go:414] "containerMap: removing container" podUID="58aba26d-82d5-4d0b-8d39-b4109b823511" containerName="maas-api" Oct 30 13:17:59.996155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.996003 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aba26d-82d5-4d0b-8d39-b4109b823511" containerName="maas-api" Oct 30 13:17:59.996155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.996041 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c884ad38-4f81-45fa-a61d-43ae56e63732" containerName="revoke-keys" Oct 30 13:17:59.996155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.996049 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c884ad38-4f81-45fa-a61d-43ae56e63732" containerName="revoke-keys" Oct 30 13:17:59.996291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.996247 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c884ad38-4f81-45fa-a61d-43ae56e63732" containerName="revoke-keys" Oct 30 13:17:59.996291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.996271 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb76201-354c-4b8c-8f6b-2483135657c6" containerName="istio-proxy" Oct 30 13:17:59.996291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:17:59.996287 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aba26d-82d5-4d0b-8d39-b4109b823511" containerName="maas-api" Oct 30 13:18:00.001255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.001226 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.024794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.024757 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9"] Oct 30 13:18:00.026770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.026734 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:18:00.026913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.026810 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:18:00.156737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.156697 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.157309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.156944 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4k8l\" (UniqueName: \"kubernetes.io/projected/2ead0126-4067-4cb4-9dc7-8b18b2215959-kube-api-access-n4k8l\") pod \"maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.157309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.157042 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.258856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.258756 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4k8l\" (UniqueName: \"kubernetes.io/projected/2ead0126-4067-4cb4-9dc7-8b18b2215959-kube-api-access-n4k8l\") pod \"maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.259037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.258866 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.259037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.258980 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.260723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.260228 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.274772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.274742 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.280611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.280587 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4k8l\" (UniqueName: \"kubernetes.io/projected/2ead0126-4067-4cb4-9dc7-8b18b2215959-kube-api-access-n4k8l\") pod \"maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.330484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.330448 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:00.649096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:00.649062 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9"] Oct 30 13:18:00.653049 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:18:00.653023 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ead0126_4067_4cb4_9dc7_8b18b2215959.slice/crio-a328fb3766843ada27788e4c07c4e4ceb0f2e6165ed7e9bbc1e1d132ecc71016 WatchSource:0}: Error finding container a328fb3766843ada27788e4c07c4e4ceb0f2e6165ed7e9bbc1e1d132ecc71016: Status 404 returned error can't find the container with id a328fb3766843ada27788e4c07c4e4ceb0f2e6165ed7e9bbc1e1d132ecc71016 Oct 30 13:18:01.617445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:01.617414 5049 generic.go:356] "Generic (PLEG): container finished" podID="2ead0126-4067-4cb4-9dc7-8b18b2215959" containerID="7c331afe792b40d447ffd8e5c49e2bf3e9153d8b5efc8d1f0d658d86ddf2cc3c" exitCode=0 Oct 30 13:18:01.617844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:01.617497 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" event={"ID":"2ead0126-4067-4cb4-9dc7-8b18b2215959","Type":"ContainerDied","Data":"7c331afe792b40d447ffd8e5c49e2bf3e9153d8b5efc8d1f0d658d86ddf2cc3c"} Oct 30 13:18:01.617844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:01.617552 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" event={"ID":"2ead0126-4067-4cb4-9dc7-8b18b2215959","Type":"ContainerStarted","Data":"a328fb3766843ada27788e4c07c4e4ceb0f2e6165ed7e9bbc1e1d132ecc71016"} Oct 30 13:18:01.799137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:01.799053 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:02.805155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:02.805123 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:02.908712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:02.908625 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ead0126-4067-4cb4-9dc7-8b18b2215959-kube-api-access-n4k8l\" (UniqueName: \"kubernetes.io/projected/2ead0126-4067-4cb4-9dc7-8b18b2215959-kube-api-access-n4k8l\") pod \"2ead0126-4067-4cb4-9dc7-8b18b2215959\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " Oct 30 13:18:02.908882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:02.908706 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-openshift-ca\") pod \"2ead0126-4067-4cb4-9dc7-8b18b2215959\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " Oct 30 13:18:02.908882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:02.908740 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-xks-ca\") pod \"2ead0126-4067-4cb4-9dc7-8b18b2215959\" (UID: \"2ead0126-4067-4cb4-9dc7-8b18b2215959\") " Oct 30 13:18:02.909155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:02.909126 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-openshift-ca" pod "2ead0126-4067-4cb4-9dc7-8b18b2215959" (UID: "2ead0126-4067-4cb4-9dc7-8b18b2215959"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:02.909460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:02.909435 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:02.910927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:02.910894 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-xks-ca" pod "2ead0126-4067-4cb4-9dc7-8b18b2215959" (UID: "2ead0126-4067-4cb4-9dc7-8b18b2215959"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:02.911027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:02.911011 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ead0126-4067-4cb4-9dc7-8b18b2215959-kube-api-access-n4k8l" pod "2ead0126-4067-4cb4-9dc7-8b18b2215959" (UID: "2ead0126-4067-4cb4-9dc7-8b18b2215959"). InnerVolumeSpecName "kube-api-access-n4k8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:03.010698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:03.010658 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4k8l\" (UniqueName: \"kubernetes.io/projected/2ead0126-4067-4cb4-9dc7-8b18b2215959-kube-api-access-n4k8l\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:03.010698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:03.010697 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ead0126-4067-4cb4-9dc7-8b18b2215959-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:03.636103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:03.636019 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" Oct 30 13:18:03.636103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:03.636051 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9" event={"ID":"2ead0126-4067-4cb4-9dc7-8b18b2215959","Type":"ContainerDied","Data":"a328fb3766843ada27788e4c07c4e4ceb0f2e6165ed7e9bbc1e1d132ecc71016"} Oct 30 13:18:03.636313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:03.636108 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a328fb3766843ada27788e4c07c4e4ceb0f2e6165ed7e9bbc1e1d132ecc71016" Oct 30 13:18:03.721341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:03.720595 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:05.702615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:05.702281 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:07.590955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:07.590915 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:09.322989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:09.322949 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:10.614196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:10.614160 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:10.780807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:10.780777 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:11.096416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:11.096342 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:11.626761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:11.626723 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:12.118294 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:12.118264 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:12.318822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:12.318788 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:12.344270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:12.344238 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt"] Oct 30 13:18:12.353576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:12.353547 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-wg5kt"] Oct 30 13:18:12.725501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:12.725427 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:12.858971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:12.858929 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm"] Oct 30 13:18:12.864884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:12.864854 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b0881-82hrm"] Oct 30 13:18:12.890724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:12.890633 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:13.343097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:13.343059 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c884ad38-4f81-45fa-a61d-43ae56e63732" path="/var/lib/kubelet/pods/c884ad38-4f81-45fa-a61d-43ae56e63732/volumes" Oct 30 13:18:13.343829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:13.343808 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbad5317-0a2d-468f-9097-d19046b8292c" path="/var/lib/kubelet/pods/cbad5317-0a2d-468f-9097-d19046b8292c/volumes" Oct 30 13:18:13.397525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:13.397492 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:14.717471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:14.717440 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:15.043667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:15.043638 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:16.894780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:16.894655 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:17.859521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:17.859491 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:18.795886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:18.795847 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:20.099795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:20.099659 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:20.258184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:20.258152 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:20.427853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:20.427754 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:20.918223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:20.918182 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:21.283311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:21.283230 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:21.875779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:21.875737 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:22.047424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:22.047043 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:22.799432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:22.799396 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:24.854863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:24.854830 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:25.766450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:25.766404 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:25.940940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:25.940902 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:26.679224 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:18:26.679186 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:18:26.995624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:26.992666 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:27.161404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.161361 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:27.428722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.428684 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:27.585268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.585221 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:27.826347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.825659 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p"] Oct 30 13:18:27.830354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.826879 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2ead0126-4067-4cb4-9dc7-8b18b2215959" containerName="revoke-keys" Oct 30 13:18:27.830354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.826905 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ead0126-4067-4cb4-9dc7-8b18b2215959" containerName="revoke-keys" Oct 30 13:18:27.830354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.827074 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ead0126-4067-4cb4-9dc7-8b18b2215959" containerName="revoke-keys" Oct 30 13:18:27.897342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.887010 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:18:27.901750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.894607 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p"] Oct 30 13:18:27.925742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.925714 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b5a369-dockercfg-ww6fm\"" Oct 30 13:18:27.925742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:27.925714 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b5a369\"" Oct 30 13:18:28.087051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.086957 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87sj\" (UniqueName: \"kubernetes.io/projected/3cd54890-4ad8-4871-affd-f78510572c06-kube-api-access-c87sj\") pod \"payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p\" (UID: \"3cd54890-4ad8-4871-affd-f78510572c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:18:28.087511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.087159 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd54890-4ad8-4871-affd-f78510572c06-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p\" (UID: \"3cd54890-4ad8-4871-affd-f78510572c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:18:28.188414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.188380 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87sj\" (UniqueName: \"kubernetes.io/projected/3cd54890-4ad8-4871-affd-f78510572c06-kube-api-access-c87sj\") pod \"payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p\" (UID: \"3cd54890-4ad8-4871-affd-f78510572c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:18:28.188592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.188526 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd54890-4ad8-4871-affd-f78510572c06-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p\" (UID: \"3cd54890-4ad8-4871-affd-f78510572c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:18:28.189277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.189255 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd54890-4ad8-4871-affd-f78510572c06-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p\" (UID: \"3cd54890-4ad8-4871-affd-f78510572c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:18:28.210582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.210554 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87sj\" (UniqueName: \"kubernetes.io/projected/3cd54890-4ad8-4871-affd-f78510572c06-kube-api-access-c87sj\") pod \"payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p\" (UID: \"3cd54890-4ad8-4871-affd-f78510572c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:18:28.226261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.226228 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:18:28.540358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.539961 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p"] Oct 30 13:18:28.543038 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:18:28.543005 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd54890_4ad8_4871_affd_f78510572c06.slice/crio-0dc4ff242857bce825398d04e3be4e59567e28024e9dadbd677d04c796081c61 WatchSource:0}: Error finding container 0dc4ff242857bce825398d04e3be4e59567e28024e9dadbd677d04c796081c61: Status 404 returned error can't find the container with id 0dc4ff242857bce825398d04e3be4e59567e28024e9dadbd677d04c796081c61 Oct 30 13:18:28.940289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.940249 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" event={"ID":"3cd54890-4ad8-4871-affd-f78510572c06","Type":"ContainerStarted","Data":"0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea"} Oct 30 13:18:28.940513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.940365 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:18:28.940513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.940393 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" event={"ID":"3cd54890-4ad8-4871-affd-f78510572c06","Type":"ContainerStarted","Data":"0dc4ff242857bce825398d04e3be4e59567e28024e9dadbd677d04c796081c61"} Oct 30 13:18:28.979071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.979017 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" podStartSLOduration=1.9790012190000001 podStartE2EDuration="1.979001219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:28.978195846 +0000 UTC m=+1615.793390272" watchObservedRunningTime="2026-10-30 13:18:28.979001219 +0000 UTC m=+1615.794195641" Oct 30 13:18:28.996594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:28.996561 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:30.989559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:30.989519 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:32.803559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:32.803514 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:34.478267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:34.478233 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:34.981204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:34.981158 5049 scope.go:122] "RemoveContainer" containerID="3e39340158cc6931739522810a28b7bbbdd9524ab4a63f9381274ca9b8088958" Oct 30 13:18:34.997828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:34.997802 5049 scope.go:122] "RemoveContainer" containerID="8059f92530acd112a570b38fcfcd31b7ceea764960b2b931e2e46d124f46a13a" Oct 30 13:18:35.049542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:35.049417 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:35.163410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:35.163377 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck"] Oct 30 13:18:35.172220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:35.172192 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fcb69-scjck"] Oct 30 13:18:35.198086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:35.198050 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:35.343995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:35.343961 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610d502a-0f69-482a-87fb-c82ed83c026d" path="/var/lib/kubelet/pods/610d502a-0f69-482a-87fb-c82ed83c026d/volumes" Oct 30 13:18:35.352236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:35.352206 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:35.739647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:35.739559 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:37.250344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:37.245876 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:39.446682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:39.446618 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:39.951568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:39.951528 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:18:41.530763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:41.528818 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:42.212138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:42.212104 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:42.377994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:42.377964 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:42.507567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:42.507464 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2"] Oct 30 13:18:42.507770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:42.507739 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" containerID="cri-o://8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc" gracePeriod=30 Oct 30 13:18:42.567076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:42.567043 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:43.296460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.296418 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66"] Oct 30 13:18:43.303255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.303228 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.328369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.324615 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66"] Oct 30 13:18:43.405660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.405617 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm66w\" (UniqueName: \"kubernetes.io/projected/50c68b4f-43a1-4c36-9493-91662b9be9ef-kube-api-access-rm66w\") pod \"maas-api-revoke-keys-e2e-api-b-84942a-ttm66\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.405880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.405856 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84942a-ttm66\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.405981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.405965 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84942a-ttm66\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.413340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.413289 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:18:43.413523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.413508 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:18:43.498110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.496001 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:43.517542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.516618 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm66w\" (UniqueName: \"kubernetes.io/projected/50c68b4f-43a1-4c36-9493-91662b9be9ef-kube-api-access-rm66w\") pod \"maas-api-revoke-keys-e2e-api-b-84942a-ttm66\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.517542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.516821 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84942a-ttm66\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.517542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.516918 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84942a-ttm66\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.527047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.520702 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84942a-ttm66\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.527047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.526874 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84942a-ttm66\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.543459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.543410 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm66w\" (UniqueName: \"kubernetes.io/projected/50c68b4f-43a1-4c36-9493-91662b9be9ef-kube-api-access-rm66w\") pod \"maas-api-revoke-keys-e2e-api-b-84942a-ttm66\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.624160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.624053 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:43.934928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:43.934878 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66"] Oct 30 13:18:43.939314 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:18:43.939286 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c68b4f_43a1_4c36_9493_91662b9be9ef.slice/crio-abbf8ef2b223b1650fc94e0666fd247346852d70fd632d0945bbf95db685896b WatchSource:0}: Error finding container abbf8ef2b223b1650fc94e0666fd247346852d70fd632d0945bbf95db685896b: Status 404 returned error can't find the container with id abbf8ef2b223b1650fc94e0666fd247346852d70fd632d0945bbf95db685896b Oct 30 13:18:44.082238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:44.082204 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" event={"ID":"50c68b4f-43a1-4c36-9493-91662b9be9ef","Type":"ContainerStarted","Data":"abbf8ef2b223b1650fc94e0666fd247346852d70fd632d0945bbf95db685896b"} Oct 30 13:18:44.321176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:44.321143 5049 patch_prober.go:29] interesting pod/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:18:44.321374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:44.321200 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:18:45.092000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:45.091970 5049 generic.go:356] "Generic (PLEG): container finished" podID="50c68b4f-43a1-4c36-9493-91662b9be9ef" containerID="a9ae8b318b266c57f5b97cc9efc0bb6427a5db91a887db245ebd272bd8d7df74" exitCode=0 Oct 30 13:18:45.092426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:45.092050 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" event={"ID":"50c68b4f-43a1-4c36-9493-91662b9be9ef","Type":"ContainerDied","Data":"a9ae8b318b266c57f5b97cc9efc0bb6427a5db91a887db245ebd272bd8d7df74"} Oct 30 13:18:45.467352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:45.465613 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:46.295627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:46.295604 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:46.388012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:46.387926 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-openshift-ca\") pod \"50c68b4f-43a1-4c36-9493-91662b9be9ef\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " Oct 30 13:18:46.388435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:46.388414 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c68b4f-43a1-4c36-9493-91662b9be9ef-kube-api-access-rm66w\" (UniqueName: \"kubernetes.io/projected/50c68b4f-43a1-4c36-9493-91662b9be9ef-kube-api-access-rm66w\") pod \"50c68b4f-43a1-4c36-9493-91662b9be9ef\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " Oct 30 13:18:46.388653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:46.388639 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-xks-ca\") pod \"50c68b4f-43a1-4c36-9493-91662b9be9ef\" (UID: \"50c68b4f-43a1-4c36-9493-91662b9be9ef\") " Oct 30 13:18:46.390062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:46.390038 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-openshift-ca" pod "50c68b4f-43a1-4c36-9493-91662b9be9ef" (UID: "50c68b4f-43a1-4c36-9493-91662b9be9ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:46.393036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:46.393006 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-xks-ca" pod "50c68b4f-43a1-4c36-9493-91662b9be9ef" (UID: "50c68b4f-43a1-4c36-9493-91662b9be9ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:46.393740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:46.393713 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c68b4f-43a1-4c36-9493-91662b9be9ef-kube-api-access-rm66w" pod "50c68b4f-43a1-4c36-9493-91662b9be9ef" (UID: "50c68b4f-43a1-4c36-9493-91662b9be9ef"). InnerVolumeSpecName "kube-api-access-rm66w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:46.491906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:46.491854 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm66w\" (UniqueName: \"kubernetes.io/projected/50c68b4f-43a1-4c36-9493-91662b9be9ef-kube-api-access-rm66w\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.492159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:46.492139 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.492266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:46.492253 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c68b4f-43a1-4c36-9493-91662b9be9ef-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:47.206789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:47.206755 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:47.208257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:47.208207 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" event={"ID":"50c68b4f-43a1-4c36-9493-91662b9be9ef","Type":"ContainerDied","Data":"abbf8ef2b223b1650fc94e0666fd247346852d70fd632d0945bbf95db685896b"} Oct 30 13:18:47.208543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:47.208523 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abbf8ef2b223b1650fc94e0666fd247346852d70fd632d0945bbf95db685896b" Oct 30 13:18:47.208663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:47.208273 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66" Oct 30 13:18:47.915350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:47.906400 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:47.974877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:47.969784 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr"] Oct 30 13:18:47.982301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:47.982029 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8af0fe-47bdr"] Oct 30 13:18:48.072951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.072925 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:48.175398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.175296 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:18:48.243802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.243767 5049 generic.go:356] "Generic (PLEG): container finished" podID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerID="8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc" exitCode=0 Oct 30 13:18:48.243802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.243811 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" Oct 30 13:18:48.244028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.243843 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" event={"ID":"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8","Type":"ContainerDied","Data":"8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc"} Oct 30 13:18:48.244028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.243880 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2" event={"ID":"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8","Type":"ContainerDied","Data":"a1a17751e0e98b2c3f53cfe52f676f49e8d828b5a019693e6b90613f90a388fa"} Oct 30 13:18:48.244028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.243903 5049 scope.go:122] "RemoveContainer" containerID="8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc" Oct 30 13:18:48.267754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.267728 5049 scope.go:122] "RemoveContainer" containerID="3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8" Oct 30 13:18:48.280432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.280404 5049 scope.go:122] "RemoveContainer" containerID="8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc" Oct 30 13:18:48.280785 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:18:48.280761 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc\": container with ID starting with 8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc not found: ID does not exist" containerID="8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc" Oct 30 13:18:48.280913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.280791 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc"} err="failed to get container status \"8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc\": rpc error: code = NotFound desc = could not find container \"8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc\": container with ID starting with 8fdc506f2c3df5b3e24fff4c5c76e3e0ccd9efa978294dea048dbb4d292ceacc not found: ID does not exist" Oct 30 13:18:48.280913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.280810 5049 scope.go:122] "RemoveContainer" containerID="3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8" Oct 30 13:18:48.281140 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:18:48.281109 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8\": container with ID starting with 3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8 not found: ID does not exist" containerID="3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8" Oct 30 13:18:48.281213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.281151 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8"} err="failed to get container status \"3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8\": rpc error: code = NotFound desc = could not find container \"3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8\": container with ID starting with 3d48620fea3858327ac533e0f6ef8c76012579eae7424c38551cbece5eced5f8 not found: ID does not exist" Oct 30 13:18:48.288209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.288182 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-credential-socket\") pod \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " Oct 30 13:18:48.288354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.288240 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-certs\") pod \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " Oct 30 13:18:48.288476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.288455 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-token\" (UniqueName: \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-token\") pod \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " Oct 30 13:18:48.288562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.288499 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-envoy\") pod \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " Oct 30 13:18:48.288562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.288494 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-credential-socket" pod "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" (UID: "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:48.288683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.288628 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-certs" pod "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" (UID: "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:48.288739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.288698 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-kube-api-access-lt2vn\" (UniqueName: \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-kube-api-access-lt2vn\") pod \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " Oct 30 13:18:48.289047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.289026 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-data\") pod \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " Oct 30 13:18:48.289183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.289166 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-podinfo\") pod \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " Oct 30 13:18:48.289345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.289302 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-data" pod "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" (UID: "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:48.289441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.289391 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istiod-ca-cert\") pod \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " Oct 30 13:18:48.289508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.289474 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-socket\") pod \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\" (UID: \"3924f6fb-a4c9-4bf7-afad-e06fe378b1b8\") " Oct 30 13:18:48.289994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.289962 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istiod-ca-cert" pod "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" (UID: "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:48.290134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.290109 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-socket" pod "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" (UID: "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:48.290543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.290517 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.290641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.290545 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.290641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.290559 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.290641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.290573 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.290641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.290586 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.291776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.291751 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-envoy" pod "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" (UID: "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:48.292026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.292004 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-token" pod "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" (UID: "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:48.292766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.292705 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-kube-api-access-lt2vn" pod "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" (UID: "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8"). InnerVolumeSpecName "kube-api-access-lt2vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:48.293411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.293378 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-podinfo" pod "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" (UID: "3924f6fb-a4c9-4bf7-afad-e06fe378b1b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:48.392906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.392826 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.392906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.392861 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.392906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.392877 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.392906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.392892 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt2vn\" (UniqueName: \"kubernetes.io/projected/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8-kube-api-access-lt2vn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.519404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.518420 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:48.600331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.600283 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2"] Oct 30 13:18:48.624477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:48.624406 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9fcb69-openshift-default-d4d876d87-z4tp2"] Oct 30 13:18:49.343682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:49.343650 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" path="/var/lib/kubelet/pods/3924f6fb-a4c9-4bf7-afad-e06fe378b1b8/volumes" Oct 30 13:18:49.344143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:49.344128 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8e7ae2-7c14-46c4-a348-d371a9a0466e" path="/var/lib/kubelet/pods/3f8e7ae2-7c14-46c4-a348-d371a9a0466e/volumes" Oct 30 13:18:50.380191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:50.379893 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:52.135442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:52.131413 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:53.718948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:53.718917 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79"] Oct 30 13:18:53.849968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:53.849939 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:54.006103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:54.006008 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:54.328344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:54.317757 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:55.963689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:55.963653 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:56.197430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:56.197396 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:56.376507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:56.369844 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:56.584360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:56.584308 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:56.678330 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:18:56.678242 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:18:57.087661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:57.086740 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:57.341492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:57.341397 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:57.539659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:57.528840 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:57.879345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:57.878097 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:58.223219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.223120 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:58.266865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.266831 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr"] Oct 30 13:18:58.267907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.267883 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" Oct 30 13:18:58.267966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.267912 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" Oct 30 13:18:58.268013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.268001 5049 cpu_manager.go:414] "containerMap: removing container" podUID="50c68b4f-43a1-4c36-9493-91662b9be9ef" containerName="revoke-keys" Oct 30 13:18:58.268048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.268013 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c68b4f-43a1-4c36-9493-91662b9be9ef" containerName="revoke-keys" Oct 30 13:18:58.268174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.268165 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" Oct 30 13:18:58.268208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.268177 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c68b4f-43a1-4c36-9493-91662b9be9ef" containerName="revoke-keys" Oct 30 13:18:58.271964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.271946 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.295221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.295189 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:18:58.296647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.296619 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:18:58.301469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.301444 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr"] Oct 30 13:18:58.336172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.336138 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4644a1-cxndr\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.336487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.336464 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4644a1-cxndr\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.336582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.336506 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54hsg\" (UniqueName: \"kubernetes.io/projected/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-kube-api-access-54hsg\") pod \"maas-api-revoke-keys-e2e-api-b-4644a1-cxndr\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.369123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.369088 5049 generic.go:356] "Generic (PLEG): container finished" podID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerID="93780e70c673bbe26c2f511141359f5e8064a8b310a8eb705ef9a38c34a6129f" exitCode=0 Oct 30 13:18:58.369264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.369162 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" event={"ID":"6eacaf2d-bdf9-4f8c-918e-44f1910a7467","Type":"ContainerDied","Data":"93780e70c673bbe26c2f511141359f5e8064a8b310a8eb705ef9a38c34a6129f"} Oct 30 13:18:58.369731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.369711 5049 scope.go:122] "RemoveContainer" containerID="93780e70c673bbe26c2f511141359f5e8064a8b310a8eb705ef9a38c34a6129f" Oct 30 13:18:58.371590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.371564 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:18:58.371684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.371628 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:18:58.371684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.371660 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:18:58.438711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.438682 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4644a1-cxndr\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.438869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.438727 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54hsg\" (UniqueName: \"kubernetes.io/projected/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-kube-api-access-54hsg\") pod \"maas-api-revoke-keys-e2e-api-b-4644a1-cxndr\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.438953 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.438929 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4644a1-cxndr\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.439684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.439651 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4644a1-cxndr\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.441708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.441678 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4644a1-cxndr\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.475698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.475629 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54hsg\" (UniqueName: \"kubernetes.io/projected/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-kube-api-access-54hsg\") pod \"maas-api-revoke-keys-e2e-api-b-4644a1-cxndr\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.586890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.586851 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:18:58.645976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.645850 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:58.737455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.736114 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" podUID="49389730-acb3-4502-ad9d-2d8ccad6ff15" containerName="payload-pre-processing" containerID="cri-o://a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27" gracePeriod=25 Oct 30 13:18:58.776672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.756385 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp"] Oct 30 13:18:58.790548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.788873 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" Oct 30 13:18:58.790548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.789160 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" Oct 30 13:18:58.791644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.790924 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3924f6fb-a4c9-4bf7-afad-e06fe378b1b8" containerName="istio-proxy" Oct 30 13:18:58.860811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.845466 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:58.860811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.850960 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp"] Oct 30 13:18:58.867293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.867073 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-31721c-openshift-default-dockercfg-h72qg\"" Oct 30 13:18:58.883904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.883873 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:58.956625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.955702 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-certs\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:58.956625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.955851 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-socket\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:58.956625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.955883 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0be57a-8d1b-4498-928c-48616b906a53-istio-podinfo\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:58.956625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.955956 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-data\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:58.956625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.956130 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-envoy\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:58.956625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.956232 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0be57a-8d1b-4498-928c-48616b906a53-istiod-ca-cert\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:58.956625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.956490 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-credential-socket\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:58.956625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.956573 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-istio-token\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:58.957065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:58.956687 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zdb\" (UniqueName: \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-kube-api-access-v2zdb\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.050405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.048552 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.059669 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-socket\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.059725 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0be57a-8d1b-4498-928c-48616b906a53-istio-podinfo\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.059760 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-data\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.059841 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-envoy\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.059918 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0be57a-8d1b-4498-928c-48616b906a53-istiod-ca-cert\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.060081 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-credential-socket\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.060135 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-istio-token\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.060198 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zdb\" (UniqueName: \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-kube-api-access-v2zdb\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.060337 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-certs\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.060813 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-certs\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.061058 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-socket\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.062096 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0be57a-8d1b-4498-928c-48616b906a53-istiod-ca-cert\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.063056 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-data\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.074351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.073040 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-credential-socket\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.078688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.077944 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0be57a-8d1b-4498-928c-48616b906a53-istio-podinfo\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.082491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.081846 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-envoy\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.160247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.160219 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr"] Oct 30 13:18:59.165793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.163544 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-istio-token\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.174773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.174645 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zdb\" (UniqueName: \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-kube-api-access-v2zdb\") pod \"e2e-inf-a-31721c-openshift-default-7685767649-hbppp\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.175919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.175895 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:18:59.280685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.280659 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:18:59.365768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.365691 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49389730-acb3-4502-ad9d-2d8ccad6ff15-kube-api-access-ffnt4\" (UniqueName: \"kubernetes.io/projected/49389730-acb3-4502-ad9d-2d8ccad6ff15-kube-api-access-ffnt4\") pod \"49389730-acb3-4502-ad9d-2d8ccad6ff15\" (UID: \"49389730-acb3-4502-ad9d-2d8ccad6ff15\") " Oct 30 13:18:59.366120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.366101 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49389730-acb3-4502-ad9d-2d8ccad6ff15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49389730-acb3-4502-ad9d-2d8ccad6ff15-plugins-config-volume\") pod \"49389730-acb3-4502-ad9d-2d8ccad6ff15\" (UID: \"49389730-acb3-4502-ad9d-2d8ccad6ff15\") " Oct 30 13:18:59.366474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.366456 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49389730-acb3-4502-ad9d-2d8ccad6ff15-plugins-config-volume" pod "49389730-acb3-4502-ad9d-2d8ccad6ff15" (UID: "49389730-acb3-4502-ad9d-2d8ccad6ff15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:59.367574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.367555 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49389730-acb3-4502-ad9d-2d8ccad6ff15-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:59.380378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.380313 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" event={"ID":"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57","Type":"ContainerStarted","Data":"0dde88e94d5b2c5d8efc346d41572c5de2ef5556f4e9e8f88155472260275bf0"} Oct 30 13:18:59.382019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.381994 5049 generic.go:356] "Generic (PLEG): container finished" podID="49389730-acb3-4502-ad9d-2d8ccad6ff15" containerID="a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27" exitCode=0 Oct 30 13:18:59.382160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.382054 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" Oct 30 13:18:59.382160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.382070 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" event={"ID":"49389730-acb3-4502-ad9d-2d8ccad6ff15","Type":"ContainerDied","Data":"a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27"} Oct 30 13:18:59.382160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.382140 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79" event={"ID":"49389730-acb3-4502-ad9d-2d8ccad6ff15","Type":"ContainerDied","Data":"741a03b8fa515153036ebf8de78e30814744703fb213a18ab58515957016d055"} Oct 30 13:18:59.382368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.382170 5049 scope.go:122] "RemoveContainer" containerID="a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27" Oct 30 13:18:59.385261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.385228 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" event={"ID":"6eacaf2d-bdf9-4f8c-918e-44f1910a7467","Type":"ContainerStarted","Data":"2c219d33a9c85ad4ae11999c8338464c86dd3fcef2010499600317f32166704b"} Oct 30 13:18:59.400649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.400624 5049 scope.go:122] "RemoveContainer" containerID="a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27" Oct 30 13:18:59.400925 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:18:59.400902 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27\": container with ID starting with a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27 not found: ID does not exist" containerID="a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27" Oct 30 13:18:59.400985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.400933 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27"} err="failed to get container status \"a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27\": rpc error: code = NotFound desc = could not find container \"a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27\": container with ID starting with a9ca67dc1e0e81f72ea5b336887fbabadb557df6c3a09f5d12b0fc49c6309a27 not found: ID does not exist" Oct 30 13:18:59.411916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.411580 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49389730-acb3-4502-ad9d-2d8ccad6ff15-kube-api-access-ffnt4" pod "49389730-acb3-4502-ad9d-2d8ccad6ff15" (UID: "49389730-acb3-4502-ad9d-2d8ccad6ff15"). InnerVolumeSpecName "kube-api-access-ffnt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:59.474262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.474132 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffnt4\" (UniqueName: \"kubernetes.io/projected/49389730-acb3-4502-ad9d-2d8ccad6ff15-kube-api-access-ffnt4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:18:59.548252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.548203 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp"] Oct 30 13:18:59.550271 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:18:59.550243 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0be57a_8d1b_4498_928c_48616b906a53.slice/crio-d3b1887537528c7f92b2369ab7e956a72fb931db0efe62a8f300fe3fd18d4e11 WatchSource:0}: Error finding container d3b1887537528c7f92b2369ab7e956a72fb931db0efe62a8f300fe3fd18d4e11: Status 404 returned error can't find the container with id d3b1887537528c7f92b2369ab7e956a72fb931db0efe62a8f300fe3fd18d4e11 Oct 30 13:18:59.554960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.554924 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:18:59.555069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.554996 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:18:59.555069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.555032 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:18:59.743654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.743591 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79"] Oct 30 13:18:59.749926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:18:59.749719 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84942a-dc9455675-t6r79"] Oct 30 13:19:00.281072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:00.281036 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:19:00.282046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:00.282024 5049 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 13:19:00.282105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:00.282067 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 13:19:00.400991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:00.400950 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" event={"ID":"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57","Type":"ContainerStarted","Data":"a7da29c23a64f277f4b3955cfe8804a01ab2901ec0d120a2bd1e0731b16c58b1"} Oct 30 13:19:00.428344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:00.405375 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" event={"ID":"eb0be57a-8d1b-4498-928c-48616b906a53","Type":"ContainerStarted","Data":"f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd"} Oct 30 13:19:00.428344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:00.405434 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" event={"ID":"eb0be57a-8d1b-4498-928c-48616b906a53","Type":"ContainerStarted","Data":"d3b1887537528c7f92b2369ab7e956a72fb931db0efe62a8f300fe3fd18d4e11"} Oct 30 13:19:00.444796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:00.444668 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" podStartSLOduration=2.444649024 podStartE2EDuration="2.444649024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:00.440984666 +0000 UTC m=+1647.256179092" watchObservedRunningTime="2026-10-30 13:19:00.444649024 +0000 UTC m=+1647.259843449" Oct 30 13:19:00.486676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:00.486625 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" podStartSLOduration=2.486610863 podStartE2EDuration="2.486610863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:00.482738166 +0000 UTC m=+1647.297932590" watchObservedRunningTime="2026-10-30 13:19:00.486610863 +0000 UTC m=+1647.301805285" Oct 30 13:19:00.800924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:00.800860 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:00.964657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:00.964618 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:01.176788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.176696 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:19:01.183099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.183071 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:19:01.281295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.281258 5049 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 13:19:01.281702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.281359 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 13:19:01.361047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.361001 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49389730-acb3-4502-ad9d-2d8ccad6ff15" path="/var/lib/kubelet/pods/49389730-acb3-4502-ad9d-2d8ccad6ff15/volumes" Oct 30 13:19:01.429724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.429629 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:01.433072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.433044 5049 generic.go:356] "Generic (PLEG): container finished" podID="ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57" containerID="a7da29c23a64f277f4b3955cfe8804a01ab2901ec0d120a2bd1e0731b16c58b1" exitCode=0 Oct 30 13:19:01.435125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.435100 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" event={"ID":"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57","Type":"ContainerDied","Data":"a7da29c23a64f277f4b3955cfe8804a01ab2901ec0d120a2bd1e0731b16c58b1"} Oct 30 13:19:01.435335 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.435301 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:19:01.439561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.439531 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:19:01.691287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:01.691184 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:02.281184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:02.281148 5049 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 13:19:02.281386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:02.281201 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 13:19:02.880120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:02.880093 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:19:03.003897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.003813 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-kube-api-access-54hsg\" (UniqueName: \"kubernetes.io/projected/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-kube-api-access-54hsg\") pod \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " Oct 30 13:19:03.004078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.004027 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-xks-ca\") pod \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " Oct 30 13:19:03.004161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.004102 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-openshift-ca\") pod \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\" (UID: \"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57\") " Oct 30 13:19:03.004684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.004649 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-openshift-ca" pod "ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57" (UID: "ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:03.004861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.004842 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.057776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.056565 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-kube-api-access-54hsg" pod "ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57" (UID: "ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57"). InnerVolumeSpecName "kube-api-access-54hsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:03.058975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.058935 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-xks-ca" pod "ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57" (UID: "ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:19:03.145347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.106537 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.145347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.106572 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54hsg\" (UniqueName: \"kubernetes.io/projected/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57-kube-api-access-54hsg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.191968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.191922 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:03.281375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.281284 5049 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 13:19:03.281730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.281689 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 13:19:03.367193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.367163 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:03.535627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.535493 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" Oct 30 13:19:03.535962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.535936 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr" event={"ID":"ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57","Type":"ContainerDied","Data":"0dde88e94d5b2c5d8efc346d41572c5de2ef5556f4e9e8f88155472260275bf0"} Oct 30 13:19:03.536105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.536090 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dde88e94d5b2c5d8efc346d41572c5de2ef5556f4e9e8f88155472260275bf0" Oct 30 13:19:03.745343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.744468 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:03.929803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:03.926234 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:04.281259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:04.281182 5049 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 13:19:04.281259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:04.281231 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 13:19:05.350342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:05.349736 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:05.353525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:05.352637 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:19:05.354344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:05.353981 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:19:05.355554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:05.355527 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:19:05.509900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:05.509867 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:05.644760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:05.644671 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5"] Oct 30 13:19:05.645311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:05.644964 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" containerID="cri-o://ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98" gracePeriod=30 Oct 30 13:19:05.751073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:05.751013 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:06.525054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:06.525018 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:06.703759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:06.703725 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:07.265827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.265789 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6"] Oct 30 13:19:07.266571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.266554 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57" containerName="revoke-keys" Oct 30 13:19:07.266571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.266572 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57" containerName="revoke-keys" Oct 30 13:19:07.266707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.266595 5049 cpu_manager.go:414] "containerMap: removing container" podUID="49389730-acb3-4502-ad9d-2d8ccad6ff15" containerName="payload-pre-processing" Oct 30 13:19:07.266707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.266601 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49389730-acb3-4502-ad9d-2d8ccad6ff15" containerName="payload-pre-processing" Oct 30 13:19:07.266773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.266743 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57" containerName="revoke-keys" Oct 30 13:19:07.266773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.266754 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="49389730-acb3-4502-ad9d-2d8ccad6ff15" containerName="payload-pre-processing" Oct 30 13:19:07.270621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.270600 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.291822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.291795 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6"] Oct 30 13:19:07.312256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.311985 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:19:07.312768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.312729 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:19:07.379524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.379485 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb670967-ng6z6\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.379788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.379774 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffrx8\" (UniqueName: \"kubernetes.io/projected/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-kube-api-access-ffrx8\") pod \"maas-api-revoke-sub-fb670967-ng6z6\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.379843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.379831 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb670967-ng6z6\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.481426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.481395 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffrx8\" (UniqueName: \"kubernetes.io/projected/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-kube-api-access-ffrx8\") pod \"maas-api-revoke-sub-fb670967-ng6z6\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.481598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.481555 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb670967-ng6z6\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.481663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.481612 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb670967-ng6z6\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.482233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.482210 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb670967-ng6z6\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.483980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.483963 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb670967-ng6z6\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.508436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.508402 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffrx8\" (UniqueName: \"kubernetes.io/projected/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-kube-api-access-ffrx8\") pod \"maas-api-revoke-sub-fb670967-ng6z6\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.585223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.585194 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:07.615551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.615503 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:07.975234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:07.975147 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6"] Oct 30 13:19:07.979390 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:19:07.979365 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebc8a98_0b2f_4952_90f9_fd71b8c17cfe.slice/crio-330508e80db32e78c83ae549b16c40ca120d3e759ad108996930e495edfd7222 WatchSource:0}: Error finding container 330508e80db32e78c83ae549b16c40ca120d3e759ad108996930e495edfd7222: Status 404 returned error can't find the container with id 330508e80db32e78c83ae549b16c40ca120d3e759ad108996930e495edfd7222 Oct 30 13:19:08.581744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:08.581707 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" event={"ID":"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe","Type":"ContainerStarted","Data":"6453ec9cf78511ef9610220e079ffba00ccb0be6092ea83d0165b7e5d673c9f7"} Oct 30 13:19:08.581943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:08.581768 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" event={"ID":"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe","Type":"ContainerStarted","Data":"330508e80db32e78c83ae549b16c40ca120d3e759ad108996930e495edfd7222"} Oct 30 13:19:08.623047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:08.622992 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" podStartSLOduration=1.62297784 podStartE2EDuration="1.62297784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:08.618263205 +0000 UTC m=+1655.433457640" watchObservedRunningTime="2026-10-30 13:19:08.62297784 +0000 UTC m=+1655.438172263" Oct 30 13:19:08.657210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:08.657173 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb"] Oct 30 13:19:08.834269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:08.834182 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:08.991502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:08.991468 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:09.258886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:09.258776 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:09.640348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:09.633889 5049 generic.go:356] "Generic (PLEG): container finished" podID="3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe" containerID="6453ec9cf78511ef9610220e079ffba00ccb0be6092ea83d0165b7e5d673c9f7" exitCode=0 Oct 30 13:19:09.640348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:09.634029 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" event={"ID":"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe","Type":"ContainerDied","Data":"6453ec9cf78511ef9610220e079ffba00ccb0be6092ea83d0165b7e5d673c9f7"} Oct 30 13:19:09.886312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:09.886255 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:09.989231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:09.989130 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9"] Oct 30 13:19:09.993132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:09.993102 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2daeae-kb8p9"] Oct 30 13:19:10.041068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.040997 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:10.550428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.550397 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:10.857742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.857698 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:10.965448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.965409 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-xks-ca\") pod \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " Oct 30 13:19:10.965700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.965674 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-openshift-ca\") pod \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " Oct 30 13:19:10.965832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.965803 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-kube-api-access-ffrx8\" (UniqueName: \"kubernetes.io/projected/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-kube-api-access-ffrx8\") pod \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\" (UID: \"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe\") " Oct 30 13:19:10.966006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.965979 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-openshift-ca" pod "3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe" (UID: "3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:10.966490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.966468 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:10.967962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.967938 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-xks-ca" pod "3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe" (UID: "3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:19:10.968095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.968073 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-kube-api-access-ffrx8" pod "3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe" (UID: "3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe"). InnerVolumeSpecName "kube-api-access-ffrx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:10.976879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:10.976856 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:19:11.067904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.067803 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-data\") pod \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " Oct 30 13:19:11.067904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.067890 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-podinfo\") pod \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " Oct 30 13:19:11.068143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.067933 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-envoy\") pod \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " Oct 30 13:19:11.068143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.068118 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-data" pod "dbb240b8-72ef-4031-aff9-c7ff4707a0aa" (UID: "dbb240b8-72ef-4031-aff9-c7ff4707a0aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:11.068466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.068440 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-token\" (UniqueName: \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-token\") pod \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " Oct 30 13:19:11.068573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.068559 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-socket\") pod \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " Oct 30 13:19:11.068966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.068684 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istiod-ca-cert\") pod \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " Oct 30 13:19:11.068966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.068773 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-kube-api-access-hrrxn\" (UniqueName: \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-kube-api-access-hrrxn\") pod \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " Oct 30 13:19:11.069139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.068983 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-credential-socket\") pod \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " Oct 30 13:19:11.069139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.069004 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-socket" pod "dbb240b8-72ef-4031-aff9-c7ff4707a0aa" (UID: "dbb240b8-72ef-4031-aff9-c7ff4707a0aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:11.069139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.069042 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-certs\") pod \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\" (UID: \"dbb240b8-72ef-4031-aff9-c7ff4707a0aa\") " Oct 30 13:19:11.069436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.069143 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istiod-ca-cert" pod "dbb240b8-72ef-4031-aff9-c7ff4707a0aa" (UID: "dbb240b8-72ef-4031-aff9-c7ff4707a0aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:11.069787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.069761 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffrx8\" (UniqueName: \"kubernetes.io/projected/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-kube-api-access-ffrx8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.069891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.069792 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.069891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.069809 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.069891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.069822 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.069891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.069823 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-credential-socket" pod "dbb240b8-72ef-4031-aff9-c7ff4707a0aa" (UID: "dbb240b8-72ef-4031-aff9-c7ff4707a0aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:11.069891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.069836 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.070222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.070019 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-certs" pod "dbb240b8-72ef-4031-aff9-c7ff4707a0aa" (UID: "dbb240b8-72ef-4031-aff9-c7ff4707a0aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:11.070567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.070546 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-podinfo" pod "dbb240b8-72ef-4031-aff9-c7ff4707a0aa" (UID: "dbb240b8-72ef-4031-aff9-c7ff4707a0aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:19:11.071886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.071860 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-token" pod "dbb240b8-72ef-4031-aff9-c7ff4707a0aa" (UID: "dbb240b8-72ef-4031-aff9-c7ff4707a0aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:11.071981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.071944 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-envoy" pod "dbb240b8-72ef-4031-aff9-c7ff4707a0aa" (UID: "dbb240b8-72ef-4031-aff9-c7ff4707a0aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:11.072467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.072439 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-kube-api-access-hrrxn" pod "dbb240b8-72ef-4031-aff9-c7ff4707a0aa" (UID: "dbb240b8-72ef-4031-aff9-c7ff4707a0aa"). InnerVolumeSpecName "kube-api-access-hrrxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:11.171128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.171089 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.171128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.171128 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.171365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.171142 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.171365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.171155 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrrxn\" (UniqueName: \"kubernetes.io/projected/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-kube-api-access-hrrxn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.171365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.171172 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.171365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.171185 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb240b8-72ef-4031-aff9-c7ff4707a0aa-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:11.338462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.338371 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ead0126-4067-4cb4-9dc7-8b18b2215959" path="/var/lib/kubelet/pods/2ead0126-4067-4cb4-9dc7-8b18b2215959/volumes" Oct 30 13:19:11.658425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.658312 5049 generic.go:356] "Generic (PLEG): container finished" podID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerID="ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98" exitCode=0 Oct 30 13:19:11.658425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.658396 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" Oct 30 13:19:11.658603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.658346 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" event={"ID":"dbb240b8-72ef-4031-aff9-c7ff4707a0aa","Type":"ContainerDied","Data":"ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98"} Oct 30 13:19:11.658603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.658484 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5" event={"ID":"dbb240b8-72ef-4031-aff9-c7ff4707a0aa","Type":"ContainerDied","Data":"779609fef91f24597d97b509075f0798c9b03ef89a35b3f2c92461bb714f3528"} Oct 30 13:19:11.658603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.658509 5049 scope.go:122] "RemoveContainer" containerID="ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98" Oct 30 13:19:11.660310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.660279 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" Oct 30 13:19:11.660514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.660351 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6" event={"ID":"3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe","Type":"ContainerDied","Data":"330508e80db32e78c83ae549b16c40ca120d3e759ad108996930e495edfd7222"} Oct 30 13:19:11.660514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.660401 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="330508e80db32e78c83ae549b16c40ca120d3e759ad108996930e495edfd7222" Oct 30 13:19:11.672312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.672293 5049 scope.go:122] "RemoveContainer" containerID="8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc" Oct 30 13:19:11.690552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.690532 5049 scope.go:122] "RemoveContainer" containerID="ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98" Oct 30 13:19:11.690870 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:19:11.690841 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98\": container with ID starting with ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98 not found: ID does not exist" containerID="ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98" Oct 30 13:19:11.690950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.690877 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98"} err="failed to get container status \"ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98\": rpc error: code = NotFound desc = could not find container \"ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98\": container with ID starting with ca61f56556c0a77e90074fd61c72cb7d7b13514deb2237a4376bb229f354ed98 not found: ID does not exist" Oct 30 13:19:11.690950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.690894 5049 scope.go:122] "RemoveContainer" containerID="8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc" Oct 30 13:19:11.691171 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:19:11.691146 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc\": container with ID starting with 8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc not found: ID does not exist" containerID="8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc" Oct 30 13:19:11.691212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.691184 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc"} err="failed to get container status \"8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc\": rpc error: code = NotFound desc = could not find container \"8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc\": container with ID starting with 8e10afb8a46fa1a38a2079cd9e61e8174d7e62954dff75b1203b14d43cf9b1dc not found: ID does not exist" Oct 30 13:19:11.719567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.719533 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5"] Oct 30 13:19:11.729406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:11.729384 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-ccjw5"] Oct 30 13:19:12.204202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:12.203608 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:13.337997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:13.337961 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" path="/var/lib/kubelet/pods/dbb240b8-72ef-4031-aff9-c7ff4707a0aa/volumes" Oct 30 13:19:13.658634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:13.658511 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" podUID="024addbd-1f62-4f03-9c83-6b597f210983" containerName="payload-processing" containerID="cri-o://f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32" gracePeriod=25 Oct 30 13:19:13.966215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:13.966190 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:19:14.014128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.014091 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024addbd-1f62-4f03-9c83-6b597f210983-kube-api-access-qx7fc\" (UniqueName: \"kubernetes.io/projected/024addbd-1f62-4f03-9c83-6b597f210983-kube-api-access-qx7fc\") pod \"024addbd-1f62-4f03-9c83-6b597f210983\" (UID: \"024addbd-1f62-4f03-9c83-6b597f210983\") " Oct 30 13:19:14.014352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.014295 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024addbd-1f62-4f03-9c83-6b597f210983-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024addbd-1f62-4f03-9c83-6b597f210983-plugins-config-volume\") pod \"024addbd-1f62-4f03-9c83-6b597f210983\" (UID: \"024addbd-1f62-4f03-9c83-6b597f210983\") " Oct 30 13:19:14.014888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.014849 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024addbd-1f62-4f03-9c83-6b597f210983-plugins-config-volume" pod "024addbd-1f62-4f03-9c83-6b597f210983" (UID: "024addbd-1f62-4f03-9c83-6b597f210983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:14.015075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.015048 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024addbd-1f62-4f03-9c83-6b597f210983-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:14.016547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.016513 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024addbd-1f62-4f03-9c83-6b597f210983-kube-api-access-qx7fc" pod "024addbd-1f62-4f03-9c83-6b597f210983" (UID: "024addbd-1f62-4f03-9c83-6b597f210983"). InnerVolumeSpecName "kube-api-access-qx7fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:14.116300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.116221 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx7fc\" (UniqueName: \"kubernetes.io/projected/024addbd-1f62-4f03-9c83-6b597f210983-kube-api-access-qx7fc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:19:14.215978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.215846 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:14.692044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.692014 5049 generic.go:356] "Generic (PLEG): container finished" podID="024addbd-1f62-4f03-9c83-6b597f210983" containerID="f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32" exitCode=0 Oct 30 13:19:14.692541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.692057 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" Oct 30 13:19:14.692541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.692095 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" event={"ID":"024addbd-1f62-4f03-9c83-6b597f210983","Type":"ContainerDied","Data":"f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32"} Oct 30 13:19:14.692541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.692157 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb" event={"ID":"024addbd-1f62-4f03-9c83-6b597f210983","Type":"ContainerDied","Data":"fb2335c7d153d8c4a269d80c26335b74655d4c860d7f3bd6dd9103e2047904f4"} Oct 30 13:19:14.692541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.692188 5049 scope.go:122] "RemoveContainer" containerID="f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32" Oct 30 13:19:14.704095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.704068 5049 scope.go:122] "RemoveContainer" containerID="f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32" Oct 30 13:19:14.704406 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:19:14.704386 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32\": container with ID starting with f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32 not found: ID does not exist" containerID="f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32" Oct 30 13:19:14.704461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.704414 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32"} err="failed to get container status \"f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32\": rpc error: code = NotFound desc = could not find container \"f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32\": container with ID starting with f18175813749edeac9128ad4439427fdbe74ec204e143eaa3c6cc5a8112aec32 not found: ID does not exist" Oct 30 13:19:14.740403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.740307 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb"] Oct 30 13:19:14.750557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:14.750528 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4644a1-787c477fd8-s5pzb"] Oct 30 13:19:15.337252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:15.337221 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024addbd-1f62-4f03-9c83-6b597f210983" path="/var/lib/kubelet/pods/024addbd-1f62-4f03-9c83-6b597f210983/volumes" Oct 30 13:19:16.009521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:16.009484 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:17.621307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:17.621260 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:17.789550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:17.789501 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:17.990064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:17.989985 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:18.176504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:18.173750 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:18.654246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:18.654213 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:18.814926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:18.814857 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:18.969579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:18.969476 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:19.389814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:19.389782 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:19.559732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:19.559687 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:20.237134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:20.237101 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:22.141082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:22.141044 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:22.906268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:22.906227 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:23.067691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:23.067657 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:24.233994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:24.233959 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:24.606512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:24.606476 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:24.774523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:24.774488 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:26.342936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.342893 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:26.479036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.479001 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p"] Oct 30 13:19:26.479861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.479840 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe" containerName="revoke-keys" Oct 30 13:19:26.479861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.479859 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe" containerName="revoke-keys" Oct 30 13:19:26.480016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.479872 5049 cpu_manager.go:414] "containerMap: removing container" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" Oct 30 13:19:26.480016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.479878 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" Oct 30 13:19:26.480016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.479930 5049 cpu_manager.go:414] "containerMap: removing container" podUID="024addbd-1f62-4f03-9c83-6b597f210983" containerName="payload-processing" Oct 30 13:19:26.480016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.479936 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024addbd-1f62-4f03-9c83-6b597f210983" containerName="payload-processing" Oct 30 13:19:26.480142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.480047 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" Oct 30 13:19:26.480142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.480058 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="024addbd-1f62-4f03-9c83-6b597f210983" containerName="payload-processing" Oct 30 13:19:26.480142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.480066 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe" containerName="revoke-keys" Oct 30 13:19:26.480142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.480072 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" Oct 30 13:19:26.484354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.484310 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:19:26.506527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.506498 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0ad384-dockercfg-rhwnd\"" Oct 30 13:19:26.507031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.507005 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p"] Oct 30 13:19:26.508574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.508545 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:26.538011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.537973 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0ad384\"" Oct 30 13:19:26.631192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.631114 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn299\" (UniqueName: \"kubernetes.io/projected/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-kube-api-access-jn299\") pod \"payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p\" (UID: \"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:19:26.631367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.631222 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p\" (UID: \"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:19:26.674340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.673160 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:26.710385 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:19:26.704905 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:19:26.739342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.738740 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn299\" (UniqueName: \"kubernetes.io/projected/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-kube-api-access-jn299\") pod \"payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p\" (UID: \"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:19:26.739342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.738846 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p\" (UID: \"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:19:26.743360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.739933 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p\" (UID: \"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:19:26.814341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:26.806021 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn299\" (UniqueName: \"kubernetes.io/projected/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-kube-api-access-jn299\") pod \"payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p\" (UID: \"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:19:27.104852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:27.104818 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:19:27.189385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:27.189350 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:27.341582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:27.341553 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:27.417693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:27.417661 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p"] Oct 30 13:19:27.418591 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:19:27.418561 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5298c6_6e5c_4513_aad1_c0c50a8b45b1.slice/crio-af68ce268eaa82112b78ef68799e42d0d135c4c6a4dfa8a2eb4ab2fdd581e6fd WatchSource:0}: Error finding container af68ce268eaa82112b78ef68799e42d0d135c4c6a4dfa8a2eb4ab2fdd581e6fd: Status 404 returned error can't find the container with id af68ce268eaa82112b78ef68799e42d0d135c4c6a4dfa8a2eb4ab2fdd581e6fd Oct 30 13:19:27.873164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:27.873129 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" event={"ID":"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1","Type":"ContainerStarted","Data":"4e1048fc18ec13addcd78e1eb9a5a87df4c10eec65933c7a15a0fa19caac8f01"} Oct 30 13:19:27.873367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:27.873181 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" event={"ID":"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1","Type":"ContainerStarted","Data":"af68ce268eaa82112b78ef68799e42d0d135c4c6a4dfa8a2eb4ab2fdd581e6fd"} Oct 30 13:19:27.873367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:27.873224 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:19:27.912196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:27.912145 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" podStartSLOduration=1.912128025 podStartE2EDuration="1.912128025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:27.909602954 +0000 UTC m=+1674.724797388" watchObservedRunningTime="2026-10-30 13:19:27.912128025 +0000 UTC m=+1674.727322449" Oct 30 13:19:28.429638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:28.427331 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:30.409709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:30.409642 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:32.232345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:32.228072 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:33.763012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.762980 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq"] Oct 30 13:19:33.763765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.763749 5049 cpu_manager.go:414] "containerMap: removing container" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" Oct 30 13:19:33.763765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.763765 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb240b8-72ef-4031-aff9-c7ff4707a0aa" containerName="istio-proxy" Oct 30 13:19:33.767311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.767293 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.780497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.780468 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ddb271-serving-cert\"" Oct 30 13:19:33.800238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.800203 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq"] Oct 30 13:19:33.856448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.856412 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rzn\" (UniqueName: \"kubernetes.io/projected/24f54e01-a626-4c94-9b8a-bb28250ceae2-kube-api-access-w2rzn\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.856671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.856652 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-maas-api-tls\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.856737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.856720 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-metrics-certs\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.856779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.856753 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f54e01-a626-4c94-9b8a-bb28250ceae2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.920492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.920456 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:33.958181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.958148 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-maas-api-tls\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.958396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.958208 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-metrics-certs\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.958396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.958232 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f54e01-a626-4c94-9b8a-bb28250ceae2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.958396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.958298 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rzn\" (UniqueName: \"kubernetes.io/projected/24f54e01-a626-4c94-9b8a-bb28250ceae2-kube-api-access-w2rzn\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.959107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.959076 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f54e01-a626-4c94-9b8a-bb28250ceae2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.960922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.960888 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-maas-api-tls\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.961034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.960931 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-metrics-certs\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:33.980848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:33.980818 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rzn\" (UniqueName: \"kubernetes.io/projected/24f54e01-a626-4c94-9b8a-bb28250ceae2-kube-api-access-w2rzn\") pod \"maas-api-e2e-inf-b-ddb271-676674db6-w4mhq\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:34.064897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:34.064861 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:34.082607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:34.082452 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:34.416229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:34.416146 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq"] Oct 30 13:19:34.419142 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:19:34.419115 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f54e01_a626_4c94_9b8a_bb28250ceae2.slice/crio-db59c76799c038718cc9c4bb12fd26660adf2bfed1f40bdeacf1edb8b5256b1f WatchSource:0}: Error finding container db59c76799c038718cc9c4bb12fd26660adf2bfed1f40bdeacf1edb8b5256b1f: Status 404 returned error can't find the container with id db59c76799c038718cc9c4bb12fd26660adf2bfed1f40bdeacf1edb8b5256b1f Oct 30 13:19:34.935898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:34.935857 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" event={"ID":"24f54e01-a626-4c94-9b8a-bb28250ceae2","Type":"ContainerStarted","Data":"52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18"} Oct 30 13:19:34.936405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:34.935948 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" event={"ID":"24f54e01-a626-4c94-9b8a-bb28250ceae2","Type":"ContainerStarted","Data":"db59c76799c038718cc9c4bb12fd26660adf2bfed1f40bdeacf1edb8b5256b1f"} Oct 30 13:19:34.936405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:34.936073 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:34.976255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:34.976206 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" podStartSLOduration=1.609393775 podStartE2EDuration="1.976192602s" totalImagesPullingTime="366.798827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:34.973873654 +0000 UTC m=+1681.789068090" watchObservedRunningTime="2026-10-30 13:19:34.976192602 +0000 UTC m=+1681.791387062" Oct 30 13:19:35.177346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:35.159849 5049 scope.go:122] "RemoveContainer" containerID="bab98d32ebc8ea4e65ee5ec2b1c8708e84a2c64f87cdaf927274e9d631c081a9" Oct 30 13:19:35.243054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:35.243025 5049 scope.go:122] "RemoveContainer" containerID="5086f0af8c526e487ae7ca7b1653fcc2c5ca8d445b3c03017b02364e3edeab7b" Oct 30 13:19:36.011458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:36.011418 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:37.787572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:37.787514 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:37.941784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:37.941750 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:38.100191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:38.100157 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:38.885754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:38.885723 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:19:39.998421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:39.998362 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:40.953074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:40.953040 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:19:42.011356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:42.004461 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:42.462758 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:42.462714 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:42.946654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:42.946613 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:43.454951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:43.454855 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:43.619826 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:43.619788 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:43.949223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:43.949187 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:45.814343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:45.808154 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:47.426204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:47.426172 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:47.581806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:47.581774 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:47.946586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:47.946536 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:49.070796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.070768 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:49.112220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.112185 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg"] Oct 30 13:19:49.119836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.119799 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:19:49.137404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.137379 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8467ed\"" Oct 30 13:19:49.163615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.163580 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg"] Oct 30 13:19:49.258346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.258024 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:49.275627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.275589 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg\" (UID: \"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:19:49.276191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.276162 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lkh\" (UniqueName: \"kubernetes.io/projected/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-kube-api-access-f4lkh\") pod \"payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg\" (UID: \"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:19:49.379304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.379214 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg\" (UID: \"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:19:49.379608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.379580 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lkh\" (UniqueName: \"kubernetes.io/projected/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-kube-api-access-f4lkh\") pod \"payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg\" (UID: \"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:19:49.380709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.380667 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg\" (UID: \"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:19:49.430175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.430141 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lkh\" (UniqueName: \"kubernetes.io/projected/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-kube-api-access-f4lkh\") pod \"payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg\" (UID: \"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:19:49.438173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.438145 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:19:49.784705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.784679 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg"] Oct 30 13:19:49.786617 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:19:49.786591 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe53f84_bd92_4ec4_9868_a9a5a1af7b6b.slice/crio-aec080aa4ae3fb149e822fa0605d2b95ee8a0294e072bc46e97f9feb19db4932 WatchSource:0}: Error finding container aec080aa4ae3fb149e822fa0605d2b95ee8a0294e072bc46e97f9feb19db4932: Status 404 returned error can't find the container with id aec080aa4ae3fb149e822fa0605d2b95ee8a0294e072bc46e97f9feb19db4932 Oct 30 13:19:49.905262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:49.905140 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:50.097874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:50.097838 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" event={"ID":"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b","Type":"ContainerStarted","Data":"1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034"} Oct 30 13:19:50.098350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:50.097909 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:19:50.098350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:50.097932 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" event={"ID":"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b","Type":"ContainerStarted","Data":"aec080aa4ae3fb149e822fa0605d2b95ee8a0294e072bc46e97f9feb19db4932"} Oct 30 13:19:50.135865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:50.135803 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" podStartSLOduration=1.135789587 podStartE2EDuration="1.135789587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:50.134032696 +0000 UTC m=+1696.949227120" watchObservedRunningTime="2026-10-30 13:19:50.135789587 +0000 UTC m=+1696.950984010" Oct 30 13:19:51.915797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:51.915758 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:53.427107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:53.427058 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:54.063259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:54.063222 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:54.137966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:54.137923 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66"] Oct 30 13:19:54.145862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:54.145831 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84942a-ttm66"] Oct 30 13:19:54.255136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:54.255098 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:54.805868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:54.805832 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:55.337659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:55.337628 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c68b4f-43a1-4c36-9493-91662b9be9ef" path="/var/lib/kubelet/pods/50c68b4f-43a1-4c36-9493-91662b9be9ef/volumes" Oct 30 13:19:55.422194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:55.422160 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb"] Oct 30 13:19:55.422438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:55.422410 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerName="istio-proxy" containerID="cri-o://f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed" gracePeriod=30 Oct 30 13:19:55.431784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:55.431756 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8"] Oct 30 13:19:55.431972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:55.431939 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" containerID="cri-o://ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798" gracePeriod=30 Oct 30 13:19:56.510804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:56.510767 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:56.679586 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:19:56.679543 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:19:58.232423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:58.232373 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:58.398203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:19:58.398164 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:00.431813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.431772 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:00.731677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.731641 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:20:00.833764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.833704 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-token\" (UniqueName: \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-token\") pod \"0b6843e1-fc62-4312-8130-019a30b2a4af\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " Oct 30 13:20:00.833972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.833825 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-kube-api-access-xkd2n\" (UniqueName: \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-kube-api-access-xkd2n\") pod \"0b6843e1-fc62-4312-8130-019a30b2a4af\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " Oct 30 13:20:00.833972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.833931 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-certs\") pod \"0b6843e1-fc62-4312-8130-019a30b2a4af\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " Oct 30 13:20:00.833972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.833962 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-envoy\") pod \"0b6843e1-fc62-4312-8130-019a30b2a4af\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " Oct 30 13:20:00.834171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.834076 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-data\") pod \"0b6843e1-fc62-4312-8130-019a30b2a4af\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " Oct 30 13:20:00.834171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.834112 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-credential-socket\") pod \"0b6843e1-fc62-4312-8130-019a30b2a4af\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " Oct 30 13:20:00.834282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.834198 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-socket\") pod \"0b6843e1-fc62-4312-8130-019a30b2a4af\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " Oct 30 13:20:00.834361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.834293 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-podinfo\") pod \"0b6843e1-fc62-4312-8130-019a30b2a4af\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " Oct 30 13:20:00.834426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.834379 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6843e1-fc62-4312-8130-019a30b2a4af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6843e1-fc62-4312-8130-019a30b2a4af-istiod-ca-cert\") pod \"0b6843e1-fc62-4312-8130-019a30b2a4af\" (UID: \"0b6843e1-fc62-4312-8130-019a30b2a4af\") " Oct 30 13:20:00.836508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.835737 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-data" pod "0b6843e1-fc62-4312-8130-019a30b2a4af" (UID: "0b6843e1-fc62-4312-8130-019a30b2a4af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:00.836508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.835951 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-credential-socket" pod "0b6843e1-fc62-4312-8130-019a30b2a4af" (UID: "0b6843e1-fc62-4312-8130-019a30b2a4af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:00.836508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.836149 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-socket" pod "0b6843e1-fc62-4312-8130-019a30b2a4af" (UID: "0b6843e1-fc62-4312-8130-019a30b2a4af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:00.836508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.836200 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6843e1-fc62-4312-8130-019a30b2a4af-istiod-ca-cert" pod "0b6843e1-fc62-4312-8130-019a30b2a4af" (UID: "0b6843e1-fc62-4312-8130-019a30b2a4af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:00.836913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.836801 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-certs" pod "0b6843e1-fc62-4312-8130-019a30b2a4af" (UID: "0b6843e1-fc62-4312-8130-019a30b2a4af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:00.841434 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.841393 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-token" pod "0b6843e1-fc62-4312-8130-019a30b2a4af" (UID: "0b6843e1-fc62-4312-8130-019a30b2a4af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:00.841546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.841454 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-envoy" pod "0b6843e1-fc62-4312-8130-019a30b2a4af" (UID: "0b6843e1-fc62-4312-8130-019a30b2a4af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:00.851707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.842636 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-kube-api-access-xkd2n" pod "0b6843e1-fc62-4312-8130-019a30b2a4af" (UID: "0b6843e1-fc62-4312-8130-019a30b2a4af"). InnerVolumeSpecName "kube-api-access-xkd2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:00.851707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.844918 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-podinfo" pod "0b6843e1-fc62-4312-8130-019a30b2a4af" (UID: "0b6843e1-fc62-4312-8130-019a30b2a4af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:20:00.936156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.936120 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:00.936156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.936152 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkd2n\" (UniqueName: \"kubernetes.io/projected/0b6843e1-fc62-4312-8130-019a30b2a4af-kube-api-access-xkd2n\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:00.936156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.936162 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:00.936472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.936171 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:00.936472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.936179 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:00.936472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.936187 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:00.936472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.936195 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6843e1-fc62-4312-8130-019a30b2a4af-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:00.936472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.936204 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6843e1-fc62-4312-8130-019a30b2a4af-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:00.936472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:00.936213 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6843e1-fc62-4312-8130-019a30b2a4af-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:01.096494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.096466 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:20:01.111763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.111734 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:20:01.207465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.207351 5049 generic.go:356] "Generic (PLEG): container finished" podID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerID="ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798" exitCode=0 Oct 30 13:20:01.207465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.207434 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" Oct 30 13:20:01.207664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.207529 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" event={"ID":"8a985f33-13d8-4e26-aa45-0709786d0fb7","Type":"ContainerDied","Data":"ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798"} Oct 30 13:20:01.207664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.207585 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8" event={"ID":"8a985f33-13d8-4e26-aa45-0709786d0fb7","Type":"ContainerDied","Data":"e26eddc2829c4d5505e5b198a56ca251aecffe6050825343b467544736f1e274"} Oct 30 13:20:01.207664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.207617 5049 scope.go:122] "RemoveContainer" containerID="ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798" Oct 30 13:20:01.210077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.210036 5049 generic.go:356] "Generic (PLEG): container finished" podID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerID="f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed" exitCode=0 Oct 30 13:20:01.210219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.210089 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" Oct 30 13:20:01.210288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.210227 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" event={"ID":"0b6843e1-fc62-4312-8130-019a30b2a4af","Type":"ContainerDied","Data":"f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed"} Oct 30 13:20:01.210288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.210277 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb" event={"ID":"0b6843e1-fc62-4312-8130-019a30b2a4af","Type":"ContainerDied","Data":"322624ec35f573383f97cd68690dbc8a52a711ad1e52387ca37268d1bf710d61"} Oct 30 13:20:01.224482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.220961 5049 scope.go:122] "RemoveContainer" containerID="ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798" Oct 30 13:20:01.224482 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:20:01.221349 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798\": container with ID starting with ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798 not found: ID does not exist" containerID="ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798" Oct 30 13:20:01.224482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.221382 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798"} err="failed to get container status \"ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798\": rpc error: code = NotFound desc = could not find container \"ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798\": container with ID starting with ec451bf75870ac0e25bac6e0840457bd9f14df3e635d686255f22a98f2f67798 not found: ID does not exist" Oct 30 13:20:01.224482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.221399 5049 scope.go:122] "RemoveContainer" containerID="f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed" Oct 30 13:20:01.239243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.239222 5049 scope.go:122] "RemoveContainer" containerID="f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed" Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.240406 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-credential-socket\") pod \"8a985f33-13d8-4e26-aa45-0709786d0fb7\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.240448 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-envoy\") pod \"8a985f33-13d8-4e26-aa45-0709786d0fb7\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.240523 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-socket\") pod \"8a985f33-13d8-4e26-aa45-0709786d0fb7\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.240600 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-data\") pod \"8a985f33-13d8-4e26-aa45-0709786d0fb7\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.240702 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-kube-api-access-5rxp7\" (UniqueName: \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-kube-api-access-5rxp7\") pod \"8a985f33-13d8-4e26-aa45-0709786d0fb7\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.240791 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a985f33-13d8-4e26-aa45-0709786d0fb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a985f33-13d8-4e26-aa45-0709786d0fb7-istiod-ca-cert\") pod \"8a985f33-13d8-4e26-aa45-0709786d0fb7\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.240947 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-certs\") pod \"8a985f33-13d8-4e26-aa45-0709786d0fb7\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.241006 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-token\" (UniqueName: \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-token\") pod \"8a985f33-13d8-4e26-aa45-0709786d0fb7\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.241097 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-podinfo\") pod \"8a985f33-13d8-4e26-aa45-0709786d0fb7\" (UID: \"8a985f33-13d8-4e26-aa45-0709786d0fb7\") " Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:20:01.242169 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed\": container with ID starting with f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed not found: ID does not exist" containerID="f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed" Oct 30 13:20:01.242495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.242202 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed"} err="failed to get container status \"f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed\": rpc error: code = NotFound desc = could not find container \"f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed\": container with ID starting with f6b28ead1b4e07a9e7a0ca21381dfe27c2c5441dea7226ed39e4f5708ad1bfed not found: ID does not exist" Oct 30 13:20:01.244204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.243868 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-credential-socket" pod "8a985f33-13d8-4e26-aa45-0709786d0fb7" (UID: "8a985f33-13d8-4e26-aa45-0709786d0fb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:01.244966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.244818 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-certs" pod "8a985f33-13d8-4e26-aa45-0709786d0fb7" (UID: "8a985f33-13d8-4e26-aa45-0709786d0fb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:01.245402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.245270 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-socket" pod "8a985f33-13d8-4e26-aa45-0709786d0fb7" (UID: "8a985f33-13d8-4e26-aa45-0709786d0fb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:01.246059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.245870 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-data" pod "8a985f33-13d8-4e26-aa45-0709786d0fb7" (UID: "8a985f33-13d8-4e26-aa45-0709786d0fb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:01.246151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.246054 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-envoy" pod "8a985f33-13d8-4e26-aa45-0709786d0fb7" (UID: "8a985f33-13d8-4e26-aa45-0709786d0fb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:01.246555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.246523 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a985f33-13d8-4e26-aa45-0709786d0fb7-istiod-ca-cert" pod "8a985f33-13d8-4e26-aa45-0709786d0fb7" (UID: "8a985f33-13d8-4e26-aa45-0709786d0fb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:01.252312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.252286 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-kube-api-access-5rxp7" pod "8a985f33-13d8-4e26-aa45-0709786d0fb7" (UID: "8a985f33-13d8-4e26-aa45-0709786d0fb7"). InnerVolumeSpecName "kube-api-access-5rxp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:01.253557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.253536 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-token" pod "8a985f33-13d8-4e26-aa45-0709786d0fb7" (UID: "8a985f33-13d8-4e26-aa45-0709786d0fb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:01.254865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.254843 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-podinfo" pod "8a985f33-13d8-4e26-aa45-0709786d0fb7" (UID: "8a985f33-13d8-4e26-aa45-0709786d0fb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:20:01.281721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.281688 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb"] Oct 30 13:20:01.288117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.288085 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-sgpxb"] Oct 30 13:20:01.343000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.342967 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:01.343000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.342996 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rxp7\" (UniqueName: \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-kube-api-access-5rxp7\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:01.343000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.343007 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a985f33-13d8-4e26-aa45-0709786d0fb7-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:01.343248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.343017 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:01.343248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.343026 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:01.343248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.343033 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:01.343248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.343041 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:01.343248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.343050 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:01.343248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.343058 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a985f33-13d8-4e26-aa45-0709786d0fb7-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:20:01.343248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.343136 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" path="/var/lib/kubelet/pods/0b6843e1-fc62-4312-8130-019a30b2a4af/volumes" Oct 30 13:20:01.555940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.555897 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8"] Oct 30 13:20:01.559984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.559955 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-7mtb8"] Oct 30 13:20:01.924111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.924026 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx"] Oct 30 13:20:01.924942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.924921 5049 cpu_manager.go:414] "containerMap: removing container" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" Oct 30 13:20:01.925040 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.924943 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" Oct 30 13:20:01.925040 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.924962 5049 cpu_manager.go:414] "containerMap: removing container" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerName="istio-proxy" Oct 30 13:20:01.925040 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.924970 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerName="istio-proxy" Oct 30 13:20:01.925182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.925134 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" containerName="istio-proxy" Oct 30 13:20:01.925182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.925147 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6843e1-fc62-4312-8130-019a30b2a4af" containerName="istio-proxy" Oct 30 13:20:01.930432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.930409 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:01.943579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.943538 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0ad384-serving-cert\"" Oct 30 13:20:01.952397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:01.952279 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx"] Oct 30 13:20:02.067873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.067838 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3557ddb3-634a-49bd-8eba-a662bcdd0bea-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.068050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.067907 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-maas-api-tls\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.068134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.068107 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcv99\" (UniqueName: \"kubernetes.io/projected/3557ddb3-634a-49bd-8eba-a662bcdd0bea-kube-api-access-fcv99\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.068232 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.068212 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-metrics-certs\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.083094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.083060 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:02.169201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.169168 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-maas-api-tls\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.169410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.169292 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcv99\" (UniqueName: \"kubernetes.io/projected/3557ddb3-634a-49bd-8eba-a662bcdd0bea-kube-api-access-fcv99\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.169410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.169395 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-metrics-certs\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.169561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.169548 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3557ddb3-634a-49bd-8eba-a662bcdd0bea-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.170429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.170398 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3557ddb3-634a-49bd-8eba-a662bcdd0bea-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.171933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.171901 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-maas-api-tls\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.172054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.171950 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-metrics-certs\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.190568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.190501 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcv99\" (UniqueName: \"kubernetes.io/projected/3557ddb3-634a-49bd-8eba-a662bcdd0bea-kube-api-access-fcv99\") pod \"maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.248346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.247199 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:02.248346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.247766 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:02.634713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.634689 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx"] Oct 30 13:20:02.638779 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:20:02.638733 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3557ddb3_634a_49bd_8eba_a662bcdd0bea.slice/crio-0d02ba9bc78d4f54b745b5abe39d052deb52f6be8791a6359264b92fccdb0294 WatchSource:0}: Error finding container 0d02ba9bc78d4f54b745b5abe39d052deb52f6be8791a6359264b92fccdb0294: Status 404 returned error can't find the container with id 0d02ba9bc78d4f54b745b5abe39d052deb52f6be8791a6359264b92fccdb0294 Oct 30 13:20:02.964962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:02.964936 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:03.134618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:03.134576 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:03.317023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:03.316983 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" event={"ID":"3557ddb3-634a-49bd-8eba-a662bcdd0bea","Type":"ContainerStarted","Data":"8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12"} Oct 30 13:20:03.317230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:03.317081 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" event={"ID":"3557ddb3-634a-49bd-8eba-a662bcdd0bea","Type":"ContainerStarted","Data":"0d02ba9bc78d4f54b745b5abe39d052deb52f6be8791a6359264b92fccdb0294"} Oct 30 13:20:03.318248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:03.318225 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:03.375344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:03.374749 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a985f33-13d8-4e26-aa45-0709786d0fb7" path="/var/lib/kubelet/pods/8a985f33-13d8-4e26-aa45-0709786d0fb7/volumes" Oct 30 13:20:03.483343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:03.476473 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:03.899056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:03.899024 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:04.154136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.153936 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:04.558618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.558578 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:04.673515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.673463 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" podStartSLOduration=3.374523818 podStartE2EDuration="3.673448725s" totalImagesPullingTime="298.924907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:03.431725673 +0000 UTC m=+1710.246920098" watchObservedRunningTime="2026-10-30 13:20:04.673448725 +0000 UTC m=+1711.488643147" Oct 30 13:20:04.674402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.674378 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr"] Oct 30 13:20:04.683933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.683908 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4644a1-cxndr"] Oct 30 13:20:04.721097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.721065 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:04.735177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.733749 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q"] Oct 30 13:20:04.739690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.739665 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.754467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.754443 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-47628d-openshift-default-dockercfg-pzxrk\"" Oct 30 13:20:04.789190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.789160 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q"] Oct 30 13:20:04.838023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.837939 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-envoy\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.838023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.837992 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-data\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.838215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.838151 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-certs\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.838267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.838213 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zp55\" (UniqueName: \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-kube-api-access-4zp55\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.838341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.838311 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-token\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.838400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.838384 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-credential-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.838447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.838424 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-podinfo\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.838497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.838451 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.838550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.838538 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istiod-ca-cert\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.880353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.880290 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:04.982345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.974456 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-token\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.982345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.974529 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-credential-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.982345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.974582 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-podinfo\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.982345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.974625 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.982345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.974689 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istiod-ca-cert\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.982345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.974839 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-envoy\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.982345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.974899 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-data\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.982345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.974998 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-certs\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.982345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.975053 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zp55\" (UniqueName: \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-kube-api-access-4zp55\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.982345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.976037 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-credential-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.990341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.984902 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-podinfo\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.990341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.985184 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-data\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.990341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.985447 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-certs\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.990341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.985687 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.990341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.986304 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istiod-ca-cert\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:04.990994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:04.990968 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-envoy\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:05.068344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:05.055825 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-token\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:05.068344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:05.066129 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zp55\" (UniqueName: \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-kube-api-access-4zp55\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:05.104439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:05.088725 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:05.283062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:05.283009 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:05.341928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:05.341888 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57" path="/var/lib/kubelet/pods/ab09a5ae-8e32-4d8b-bc17-3c2d2e02ca57/volumes" Oct 30 13:20:05.364502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:05.364421 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:05.712351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:05.712252 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q"] Oct 30 13:20:05.716613 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:20:05.716586 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a94e7e5_3c0e_4f97_a16c_622e809fcd71.slice/crio-06c7916bf1d50ebcc22cbdf1999e344cd71bc4bb08eb21c9c25a14b9e7c83818 WatchSource:0}: Error finding container 06c7916bf1d50ebcc22cbdf1999e344cd71bc4bb08eb21c9c25a14b9e7c83818: Status 404 returned error can't find the container with id 06c7916bf1d50ebcc22cbdf1999e344cd71bc4bb08eb21c9c25a14b9e7c83818 Oct 30 13:20:05.721773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:05.721708 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:05.722174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:05.722146 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:05.722753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:05.722705 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:06.425580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:06.425539 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" event={"ID":"1a94e7e5-3c0e-4f97-a16c-622e809fcd71","Type":"ContainerStarted","Data":"795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c"} Oct 30 13:20:06.426023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:06.425651 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" event={"ID":"1a94e7e5-3c0e-4f97-a16c-622e809fcd71","Type":"ContainerStarted","Data":"06c7916bf1d50ebcc22cbdf1999e344cd71bc4bb08eb21c9c25a14b9e7c83818"} Oct 30 13:20:06.484021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:06.483970 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" podStartSLOduration=2.4839569519999998 podStartE2EDuration="2.483956952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:06.480058579 +0000 UTC m=+1713.295253012" watchObservedRunningTime="2026-10-30 13:20:06.483956952 +0000 UTC m=+1713.299151375" Oct 30 13:20:07.073248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:07.064939 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:07.365048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:07.364937 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:07.376380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:07.376346 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:07.436162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:07.436123 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:07.437227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:07.437205 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:20:09.000120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:09.000086 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:09.203342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:09.200801 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:10.423973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:10.423945 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:20:10.942246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:10.942211 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:11.137665 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:11.137618 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:11.511374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:11.509859 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:13.320455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:13.320422 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:13.677822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:13.677738 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:13.862346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:13.855086 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:14.521527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:14.521487 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:15.069345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:15.050306 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:15.307153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:15.307057 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:15.618821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:15.618760 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:17.550593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:17.550478 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:17.978249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:17.978146 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:19.619333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:19.619286 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:19.930401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:19.923096 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:21.409840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:21.409807 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:21.584282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:21.584252 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:21.991818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:21.991787 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:23.886382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:23.885556 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:24.913805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:24.913768 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:25.470128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:25.470093 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:25.870472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:25.870437 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:26.074348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:26.073800 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:26.679569 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:20:26.679530 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:20:27.326750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:27.326718 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:27.795215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:27.795180 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:29.326191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:29.326145 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:29.495700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:29.495666 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:29.755264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:29.755169 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:29.944005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:29.943962 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:31.142374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.138112 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:31.334884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.334843 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:31.383702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.383670 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj"] Oct 30 13:20:31.389082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.389055 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:20:31.418993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.418953 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj"] Oct 30 13:20:31.430311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.423309 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5dd8cc-dockercfg-kkm24\"" Oct 30 13:20:31.430311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.423413 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5dd8cc\"" Oct 30 13:20:31.839453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.839414 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzrr\" (UniqueName: \"kubernetes.io/projected/24cfc009-e002-4af1-b4a0-13b7bcf299f3-kube-api-access-8dzrr\") pod \"payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj\" (UID: \"24cfc009-e002-4af1-b4a0-13b7bcf299f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:20:31.839694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.839578 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cfc009-e002-4af1-b4a0-13b7bcf299f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj\" (UID: \"24cfc009-e002-4af1-b4a0-13b7bcf299f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:20:31.941425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.941375 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzrr\" (UniqueName: \"kubernetes.io/projected/24cfc009-e002-4af1-b4a0-13b7bcf299f3-kube-api-access-8dzrr\") pod \"payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj\" (UID: \"24cfc009-e002-4af1-b4a0-13b7bcf299f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:20:31.941613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.941537 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cfc009-e002-4af1-b4a0-13b7bcf299f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj\" (UID: \"24cfc009-e002-4af1-b4a0-13b7bcf299f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:20:31.942257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.942233 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cfc009-e002-4af1-b4a0-13b7bcf299f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj\" (UID: \"24cfc009-e002-4af1-b4a0-13b7bcf299f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:20:31.963850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:31.963815 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzrr\" (UniqueName: \"kubernetes.io/projected/24cfc009-e002-4af1-b4a0-13b7bcf299f3-kube-api-access-8dzrr\") pod \"payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj\" (UID: \"24cfc009-e002-4af1-b4a0-13b7bcf299f3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:20:32.005791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:32.005754 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:20:32.138564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:32.127552 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:32.370112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:32.370082 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj"] Oct 30 13:20:32.373116 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:20:32.373090 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cfc009_e002_4af1_b4a0_13b7bcf299f3.slice/crio-93063e53d58f4775acbb8547098d63e63f2c079ae4b9b97911f0aaa718430bd9 WatchSource:0}: Error finding container 93063e53d58f4775acbb8547098d63e63f2c079ae4b9b97911f0aaa718430bd9: Status 404 returned error can't find the container with id 93063e53d58f4775acbb8547098d63e63f2c079ae4b9b97911f0aaa718430bd9 Oct 30 13:20:32.866344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:32.866084 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" event={"ID":"24cfc009-e002-4af1-b4a0-13b7bcf299f3","Type":"ContainerStarted","Data":"c4b1b31119d9e435945f383cb60c7ea34050f02b0db8d4786798bfaa45ec2978"} Oct 30 13:20:32.866344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:32.866158 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" event={"ID":"24cfc009-e002-4af1-b4a0-13b7bcf299f3","Type":"ContainerStarted","Data":"93063e53d58f4775acbb8547098d63e63f2c079ae4b9b97911f0aaa718430bd9"} Oct 30 13:20:32.907347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:32.886966 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:20:34.030344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:34.029610 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:35.196198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:35.196162 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:35.377613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:35.377583 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:35.992958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:35.992922 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:37.814661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:37.814575 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:39.542729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:39.542635 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:41.399490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:41.399452 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:42.987443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:42.986304 5049 generic.go:356] "Generic (PLEG): container finished" podID="eb0be57a-8d1b-4498-928c-48616b906a53" containerID="f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd" exitCode=0 Oct 30 13:20:42.987443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:42.986407 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" event={"ID":"eb0be57a-8d1b-4498-928c-48616b906a53","Type":"ContainerDied","Data":"f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd"} Oct 30 13:20:42.987443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:42.987029 5049 scope.go:122] "RemoveContainer" containerID="f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd" Oct 30 13:20:42.991574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:42.991533 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:42.991816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:42.991794 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:42.991981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:42.991964 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:43.132342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:43.130454 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" podStartSLOduration=12.130433601 podStartE2EDuration="12.130433601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:32.954595432 +0000 UTC m=+1739.769789856" watchObservedRunningTime="2026-10-30 13:20:43.130433601 +0000 UTC m=+1749.945628024" Oct 30 13:20:43.294002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:43.293962 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:43.996832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:43.996795 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" event={"ID":"eb0be57a-8d1b-4498-928c-48616b906a53","Type":"ContainerStarted","Data":"3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e"} Oct 30 13:20:44.177394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:44.177272 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:20:44.178482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:44.178454 5049 patch_prober.go:29] interesting pod/e2e-inf-a-31721c-openshift-default-7685767649-hbppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:20:44.178632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:44.178518 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 13:20:44.764959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:44.764917 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:44.888436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:44.888408 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:20:45.177500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:45.177463 5049 patch_prober.go:29] interesting pod/e2e-inf-a-31721c-openshift-default-7685767649-hbppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:20:45.177896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:45.177525 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 13:20:45.303766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:45.303720 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:45.516406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:45.510241 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:45.842036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:45.841952 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:46.180245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:46.180156 5049 patch_prober.go:29] interesting pod/e2e-inf-a-31721c-openshift-default-7685767649-hbppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:20:46.180715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:46.180235 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 13:20:46.437532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:46.437438 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:20:46.457381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:46.457265 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:47.177641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:47.177607 5049 patch_prober.go:29] interesting pod/e2e-inf-a-31721c-openshift-default-7685767649-hbppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:20:47.177827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:47.177665 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 13:20:47.329783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:47.329747 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:47.523448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:47.514715 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:47.955775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:47.955728 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:48.177338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:48.177290 5049 patch_prober.go:29] interesting pod/e2e-inf-a-31721c-openshift-default-7685767649-hbppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:20:48.177540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:48.177384 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 13:20:49.177256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:49.177212 5049 patch_prober.go:29] interesting pod/e2e-inf-a-31721c-openshift-default-7685767649-hbppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 13:20:49.177684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:49.177277 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 13:20:49.903431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:49.898538 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:50.182724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:50.182654 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:20:50.183811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:50.183789 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:20:50.354922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:50.351416 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:50.539649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:50.539566 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:51.074926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:51.074886 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:51.548160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:51.548125 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:51.714385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:51.714348 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:51.987248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:51.987166 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6"] Oct 30 13:20:51.992411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:51.992388 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.010091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.010054 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-09e4c0-openshift-default-dockercfg-vp2vx\"" Oct 30 13:20:52.030363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.030330 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6"] Oct 30 13:20:52.205349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.204974 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-data\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.205349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.205086 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-podinfo\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.205349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.205128 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-token\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.205349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.205275 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-credential-socket\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.205686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.205412 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-certs\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.205686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.205465 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-envoy\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.205686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.205526 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxx6t\" (UniqueName: \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-kube-api-access-bxx6t\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.205686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.205642 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istiod-ca-cert\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.205686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.205676 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-socket\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.239393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.238969 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:52.309404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.309364 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-data\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.322700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.322665 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-podinfo\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.322938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.322923 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-token\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.323157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.323144 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-credential-socket\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.323419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.323400 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-certs\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.323665 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.323634 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-envoy\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.323819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.323804 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxx6t\" (UniqueName: \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-kube-api-access-bxx6t\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.324003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.323987 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istiod-ca-cert\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.324112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.324100 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-socket\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.325022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.324999 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-socket\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.325166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.310212 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-data\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.326493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.326469 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-certs\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.333862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.332687 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-credential-socket\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.334031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.333291 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istiod-ca-cert\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.362451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.362414 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-podinfo\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.362676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.362484 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-token\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.368449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.368307 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxx6t\" (UniqueName: \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-kube-api-access-bxx6t\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.371851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.371767 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-envoy\") pod \"e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.395715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.395657 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk"] Oct 30 13:20:52.404343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.403934 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.431347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.421190 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-47628d-serving-cert\"" Oct 30 13:20:52.431347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.426988 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.431347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.427126 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08571487-c812-4d29-9dca-7e20a422a2ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.431347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.427156 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-metrics-certs\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.431347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.427239 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8vn\" (UniqueName: \"kubernetes.io/projected/08571487-c812-4d29-9dca-7e20a422a2ac-kube-api-access-ks8vn\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.431347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.429558 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk"] Oct 30 13:20:52.529217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.529134 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.529511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.529482 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08571487-c812-4d29-9dca-7e20a422a2ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.529643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.529517 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-metrics-certs\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.529643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.529576 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8vn\" (UniqueName: \"kubernetes.io/projected/08571487-c812-4d29-9dca-7e20a422a2ac-kube-api-access-ks8vn\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.530412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.530379 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08571487-c812-4d29-9dca-7e20a422a2ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.537768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.537750 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.537896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.537815 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-metrics-certs\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.542033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.541960 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:52.557807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.557782 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8vn\" (UniqueName: \"kubernetes.io/projected/08571487-c812-4d29-9dca-7e20a422a2ac-kube-api-access-ks8vn\") pod \"maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.626267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.626235 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:52.688132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.688039 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:52.755351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.753013 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:52.837345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.837017 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:52.958942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.958907 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6"] Oct 30 13:20:52.961446 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:20:52.961417 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e9360b_c35c_47bd_b0fe_1ccad4274ab9.slice/crio-7ec5e48ce3404c16bf87580932fb675d26e3ee65ca57a22fa894b7c96fdf7256 WatchSource:0}: Error finding container 7ec5e48ce3404c16bf87580932fb675d26e3ee65ca57a22fa894b7c96fdf7256: Status 404 returned error can't find the container with id 7ec5e48ce3404c16bf87580932fb675d26e3ee65ca57a22fa894b7c96fdf7256 Oct 30 13:20:52.964995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.964961 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:52.965074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.965027 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:52.965074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:52.965056 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:53.145012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:53.144870 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk"] Oct 30 13:20:53.149555 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:20:53.149526 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08571487_c812_4d29_9dca_7e20a422a2ac.slice/crio-2e2e63b18538c3528e14c1ab956d120265595cdef2be6d145110929a4a4eff82 WatchSource:0}: Error finding container 2e2e63b18538c3528e14c1ab956d120265595cdef2be6d145110929a4a4eff82: Status 404 returned error can't find the container with id 2e2e63b18538c3528e14c1ab956d120265595cdef2be6d145110929a4a4eff82 Oct 30 13:20:53.289989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:53.289948 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" event={"ID":"38e9360b-c35c-47bd-b0fe-1ccad4274ab9","Type":"ContainerStarted","Data":"29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c"} Oct 30 13:20:53.290207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:53.290017 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" event={"ID":"38e9360b-c35c-47bd-b0fe-1ccad4274ab9","Type":"ContainerStarted","Data":"7ec5e48ce3404c16bf87580932fb675d26e3ee65ca57a22fa894b7c96fdf7256"} Oct 30 13:20:53.291736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:53.291710 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" event={"ID":"08571487-c812-4d29-9dca-7e20a422a2ac","Type":"ContainerStarted","Data":"2e2e63b18538c3528e14c1ab956d120265595cdef2be6d145110929a4a4eff82"} Oct 30 13:20:53.626860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:53.626833 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:54.303837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:54.303791 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" event={"ID":"08571487-c812-4d29-9dca-7e20a422a2ac","Type":"ContainerStarted","Data":"fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19"} Oct 30 13:20:54.304277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:54.304257 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:20:54.343049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:54.343003 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" podStartSLOduration=1.908440519 podStartE2EDuration="2.342989181s" totalImagesPullingTime="434.548662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:54.341412191 +0000 UTC m=+1761.156606614" watchObservedRunningTime="2026-10-30 13:20:54.342989181 +0000 UTC m=+1761.158183645" Oct 30 13:20:54.344902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:54.344864 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" podStartSLOduration=3.344853999 podStartE2EDuration="3.344853999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:53.366878799 +0000 UTC m=+1760.182073223" watchObservedRunningTime="2026-10-30 13:20:54.344853999 +0000 UTC m=+1761.160048423" Oct 30 13:20:54.503431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:54.499870 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:54.635462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:54.635382 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:55.314282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.314232 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:55.315297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.315272 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:20:55.634151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.634071 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:55.699998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.699959 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp"] Oct 30 13:20:55.705108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.705087 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:20:55.739903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.739872 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-8d82d5\"" Oct 30 13:20:55.745532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.745501 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp"] Oct 30 13:20:55.755160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.755123 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e483618c-00a8-458a-b8e8-bc927c60ba52-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp\" (UID: \"e483618c-00a8-458a-b8e8-bc927c60ba52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:20:55.755435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.755415 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvkbw\" (UniqueName: \"kubernetes.io/projected/e483618c-00a8-458a-b8e8-bc927c60ba52-kube-api-access-hvkbw\") pod \"payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp\" (UID: \"e483618c-00a8-458a-b8e8-bc927c60ba52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:20:55.859881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.859621 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvkbw\" (UniqueName: \"kubernetes.io/projected/e483618c-00a8-458a-b8e8-bc927c60ba52-kube-api-access-hvkbw\") pod \"payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp\" (UID: \"e483618c-00a8-458a-b8e8-bc927c60ba52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:20:55.875340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.865898 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:55.875340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.872692 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e483618c-00a8-458a-b8e8-bc927c60ba52-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp\" (UID: \"e483618c-00a8-458a-b8e8-bc927c60ba52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:20:55.875340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:55.873972 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e483618c-00a8-458a-b8e8-bc927c60ba52-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp\" (UID: \"e483618c-00a8-458a-b8e8-bc927c60ba52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:20:56.037413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:56.031205 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvkbw\" (UniqueName: \"kubernetes.io/projected/e483618c-00a8-458a-b8e8-bc927c60ba52-kube-api-access-hvkbw\") pod \"payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp\" (UID: \"e483618c-00a8-458a-b8e8-bc927c60ba52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:20:56.142356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:56.133182 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:56.315743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:56.315700 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:56.320439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:56.320413 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:20:56.643821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:56.643795 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp"] Oct 30 13:20:56.646958 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:20:56.646922 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode483618c_00a8_458a_b8e8_bc927c60ba52.slice/crio-68d36e9de9045b6f64b5f801eab7d63eedf937763bc760843b3cbd6ee3aa03ec WatchSource:0}: Error finding container 68d36e9de9045b6f64b5f801eab7d63eedf937763bc760843b3cbd6ee3aa03ec: Status 404 returned error can't find the container with id 68d36e9de9045b6f64b5f801eab7d63eedf937763bc760843b3cbd6ee3aa03ec Oct 30 13:20:56.681003 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:20:56.680973 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:20:56.810260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:56.808969 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:57.339243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:57.339209 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:20:57.339463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:57.339284 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" event={"ID":"e483618c-00a8-458a-b8e8-bc927c60ba52","Type":"ContainerStarted","Data":"348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47"} Oct 30 13:20:57.339463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:57.339335 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" event={"ID":"e483618c-00a8-458a-b8e8-bc927c60ba52","Type":"ContainerStarted","Data":"68d36e9de9045b6f64b5f801eab7d63eedf937763bc760843b3cbd6ee3aa03ec"} Oct 30 13:20:57.373211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:57.373152 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" podStartSLOduration=2.373137458 podStartE2EDuration="2.373137458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:57.371285841 +0000 UTC m=+1764.186480264" watchObservedRunningTime="2026-10-30 13:20:57.373137458 +0000 UTC m=+1764.188331896" Oct 30 13:20:58.759371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:58.751995 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:58.971470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:58.971438 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:59.124364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:20:59.124300 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:00.339102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:00.339073 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:21:00.711465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:00.704205 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:02.596446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:02.596407 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:04.460910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:04.460876 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:04.621840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:04.621806 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:06.489901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:06.489867 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:08.345095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:08.345062 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:21:08.385165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:08.382400 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:09.913830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:09.913793 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:10.083615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:10.083578 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:10.239881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:10.239791 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:11.049456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:11.049416 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb"] Oct 30 13:21:11.049939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:11.049652 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" containerID="cri-o://c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6" gracePeriod=30 Oct 30 13:21:11.339903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:11.339816 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m"] Oct 30 13:21:11.340278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:11.340243 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" containerID="cri-o://0aa25785191760e8c0dc85f46d8d8f26d0eb5d69731190d44934585eecb1c896" gracePeriod=30 Oct 30 13:21:12.060715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:12.060680 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:13.890545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:13.890507 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:15.108103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:15.108058 5049 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:21:15.108544 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:15.108129 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:21:15.157933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:15.157897 5049 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:21:15.158107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:15.157963 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:21:15.866386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:15.866293 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:16.022222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.022177 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:16.198679 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:21:16.198643 5049 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e20740_a55b_4ca5_8f06_f01b8b9cd57d.slice/crio-conmon-c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:21:16.485567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.485533 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:21:16.552612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.552568 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-socket\") pod \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " Oct 30 13:21:16.552809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.552627 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-data\") pod \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " Oct 30 13:21:16.552809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.552666 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istiod-ca-cert\") pod \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " Oct 30 13:21:16.552932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.552909 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-credential-socket\") pod \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " Oct 30 13:21:16.553097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.553058 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-token\" (UniqueName: \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-token\") pod \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " Oct 30 13:21:16.553263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.553243 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-certs\") pod \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " Oct 30 13:21:16.553381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.553316 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-podinfo\") pod \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " Oct 30 13:21:16.553442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.553415 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-envoy\") pod \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " Oct 30 13:21:16.553504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.553474 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-kube-api-access-qpqfw\" (UniqueName: \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-kube-api-access-qpqfw\") pod \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\" (UID: \"06e20740-a55b-4ca5-8f06-f01b8b9cd57d\") " Oct 30 13:21:16.556873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.555279 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-data" pod "06e20740-a55b-4ca5-8f06-f01b8b9cd57d" (UID: "06e20740-a55b-4ca5-8f06-f01b8b9cd57d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:16.556873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.555491 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istiod-ca-cert" pod "06e20740-a55b-4ca5-8f06-f01b8b9cd57d" (UID: "06e20740-a55b-4ca5-8f06-f01b8b9cd57d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:16.556873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.555949 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-certs" pod "06e20740-a55b-4ca5-8f06-f01b8b9cd57d" (UID: "06e20740-a55b-4ca5-8f06-f01b8b9cd57d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:16.556873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.556290 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-socket" pod "06e20740-a55b-4ca5-8f06-f01b8b9cd57d" (UID: "06e20740-a55b-4ca5-8f06-f01b8b9cd57d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:16.556873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.556715 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-credential-socket" pod "06e20740-a55b-4ca5-8f06-f01b8b9cd57d" (UID: "06e20740-a55b-4ca5-8f06-f01b8b9cd57d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:16.559736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.559709 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-envoy" pod "06e20740-a55b-4ca5-8f06-f01b8b9cd57d" (UID: "06e20740-a55b-4ca5-8f06-f01b8b9cd57d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:16.561625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.561599 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-podinfo" pod "06e20740-a55b-4ca5-8f06-f01b8b9cd57d" (UID: "06e20740-a55b-4ca5-8f06-f01b8b9cd57d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:21:16.562156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.562130 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-kube-api-access-qpqfw" pod "06e20740-a55b-4ca5-8f06-f01b8b9cd57d" (UID: "06e20740-a55b-4ca5-8f06-f01b8b9cd57d"). InnerVolumeSpecName "kube-api-access-qpqfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:16.567439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.566237 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-token" pod "06e20740-a55b-4ca5-8f06-f01b8b9cd57d" (UID: "06e20740-a55b-4ca5-8f06-f01b8b9cd57d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:16.655917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.655717 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.655917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.655750 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.655917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.655764 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.655917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.655780 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.655917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.655794 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.655917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.655807 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.655917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.655820 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.655917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.655832 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.655917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.655844 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpqfw\" (UniqueName: \"kubernetes.io/projected/06e20740-a55b-4ca5-8f06-f01b8b9cd57d-kube-api-access-qpqfw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.671466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.669671 5049 generic.go:356] "Generic (PLEG): container finished" podID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerID="c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6" exitCode=0 Oct 30 13:21:16.671466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.669991 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" event={"ID":"06e20740-a55b-4ca5-8f06-f01b8b9cd57d","Type":"ContainerDied","Data":"c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6"} Oct 30 13:21:16.671466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.670124 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" event={"ID":"06e20740-a55b-4ca5-8f06-f01b8b9cd57d","Type":"ContainerDied","Data":"e4b08914b1904de33fa1bf771cb65f650c8971fb7ebceebf16f38148f003ac81"} Oct 30 13:21:16.671466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.670160 5049 scope.go:122] "RemoveContainer" containerID="c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6" Oct 30 13:21:16.671466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.671023 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb" Oct 30 13:21:16.681503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.680744 5049 generic.go:356] "Generic (PLEG): container finished" podID="87855fb7-df0f-468b-9b00-55f680e3a242" containerID="0aa25785191760e8c0dc85f46d8d8f26d0eb5d69731190d44934585eecb1c896" exitCode=0 Oct 30 13:21:16.681503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.680908 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" event={"ID":"87855fb7-df0f-468b-9b00-55f680e3a242","Type":"ContainerDied","Data":"0aa25785191760e8c0dc85f46d8d8f26d0eb5d69731190d44934585eecb1c896"} Oct 30 13:21:16.780415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.780382 5049 scope.go:122] "RemoveContainer" containerID="5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041" Oct 30 13:21:16.786610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.786552 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb"] Oct 30 13:21:16.799884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.799848 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-z8tfb"] Oct 30 13:21:16.862477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.862457 5049 scope.go:122] "RemoveContainer" containerID="c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6" Oct 30 13:21:16.863941 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:21:16.863907 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6\": container with ID starting with c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6 not found: ID does not exist" containerID="c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6" Oct 30 13:21:16.864052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.863953 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6"} err="failed to get container status \"c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6\": rpc error: code = NotFound desc = could not find container \"c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6\": container with ID starting with c37d13d1835c035c0aa11ae9ff84b2ef8a2cf8fdec9ba63564c982c1308fb9b6 not found: ID does not exist" Oct 30 13:21:16.864052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.863981 5049 scope.go:122] "RemoveContainer" containerID="5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041" Oct 30 13:21:16.868673 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:21:16.868640 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041\": container with ID starting with 5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041 not found: ID does not exist" containerID="5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041" Oct 30 13:21:16.868791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.868687 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041"} err="failed to get container status \"5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041\": rpc error: code = NotFound desc = could not find container \"5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041\": container with ID starting with 5e69457ce4bff3d9d5addeb92e1800331c5be9f8ace61e62533bfd7131d5b041 not found: ID does not exist" Oct 30 13:21:16.868791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.868712 5049 scope.go:122] "RemoveContainer" containerID="17a2c00c77578ef6aea3e7dcb338b55cfabf690e3bb4ecea302b29b97a116958" Oct 30 13:21:16.878460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.878439 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:21:16.886671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.886651 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87855fb7-df0f-468b-9b00-55f680e3a242-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87855fb7-df0f-468b-9b00-55f680e3a242-istiod-ca-cert\") pod \"87855fb7-df0f-468b-9b00-55f680e3a242\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " Oct 30 13:21:16.886742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.886730 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-kube-api-access-xbcmg\" (UniqueName: \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-kube-api-access-xbcmg\") pod \"87855fb7-df0f-468b-9b00-55f680e3a242\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " Oct 30 13:21:16.886787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.886777 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-envoy\") pod \"87855fb7-df0f-468b-9b00-55f680e3a242\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " Oct 30 13:21:16.886906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.886890 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-certs\") pod \"87855fb7-df0f-468b-9b00-55f680e3a242\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " Oct 30 13:21:16.886957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.886933 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-data\") pod \"87855fb7-df0f-468b-9b00-55f680e3a242\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " Oct 30 13:21:16.887029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.886977 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87855fb7-df0f-468b-9b00-55f680e3a242-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87855fb7-df0f-468b-9b00-55f680e3a242-istio-podinfo\") pod \"87855fb7-df0f-468b-9b00-55f680e3a242\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " Oct 30 13:21:16.887085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.887017 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87855fb7-df0f-468b-9b00-55f680e3a242-istiod-ca-cert" pod "87855fb7-df0f-468b-9b00-55f680e3a242" (UID: "87855fb7-df0f-468b-9b00-55f680e3a242"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:16.887145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.887103 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-istio-token\" (UniqueName: \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-istio-token\") pod \"87855fb7-df0f-468b-9b00-55f680e3a242\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " Oct 30 13:21:16.887203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.887162 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-certs" pod "87855fb7-df0f-468b-9b00-55f680e3a242" (UID: "87855fb7-df0f-468b-9b00-55f680e3a242"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:16.887203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.887166 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-credential-socket\") pod \"87855fb7-df0f-468b-9b00-55f680e3a242\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " Oct 30 13:21:16.887315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.887247 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-socket\") pod \"87855fb7-df0f-468b-9b00-55f680e3a242\" (UID: \"87855fb7-df0f-468b-9b00-55f680e3a242\") " Oct 30 13:21:16.887315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.887270 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-data" pod "87855fb7-df0f-468b-9b00-55f680e3a242" (UID: "87855fb7-df0f-468b-9b00-55f680e3a242"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:16.887654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.887627 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-socket" pod "87855fb7-df0f-468b-9b00-55f680e3a242" (UID: "87855fb7-df0f-468b-9b00-55f680e3a242"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:16.887654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.887649 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-credential-socket" pod "87855fb7-df0f-468b-9b00-55f680e3a242" (UID: "87855fb7-df0f-468b-9b00-55f680e3a242"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:16.888046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.888028 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.888046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.888047 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.888186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.888057 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.888186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.888066 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87855fb7-df0f-468b-9b00-55f680e3a242-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.888186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.888076 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.889530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.889503 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-istio-token" pod "87855fb7-df0f-468b-9b00-55f680e3a242" (UID: "87855fb7-df0f-468b-9b00-55f680e3a242"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:16.889667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.889651 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-envoy" pod "87855fb7-df0f-468b-9b00-55f680e3a242" (UID: "87855fb7-df0f-468b-9b00-55f680e3a242"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:16.889867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.889848 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87855fb7-df0f-468b-9b00-55f680e3a242-istio-podinfo" pod "87855fb7-df0f-468b-9b00-55f680e3a242" (UID: "87855fb7-df0f-468b-9b00-55f680e3a242"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:21:16.889903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.889868 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-kube-api-access-xbcmg" pod "87855fb7-df0f-468b-9b00-55f680e3a242" (UID: "87855fb7-df0f-468b-9b00-55f680e3a242"). InnerVolumeSpecName "kube-api-access-xbcmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:16.990221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.990129 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87855fb7-df0f-468b-9b00-55f680e3a242-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.990221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.990169 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87855fb7-df0f-468b-9b00-55f680e3a242-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.990221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.990183 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:16.990221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:16.990196 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbcmg\" (UniqueName: \"kubernetes.io/projected/87855fb7-df0f-468b-9b00-55f680e3a242-kube-api-access-xbcmg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:17.352627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:17.352592 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" path="/var/lib/kubelet/pods/06e20740-a55b-4ca5-8f06-f01b8b9cd57d/volumes" Oct 30 13:21:17.693851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:17.693771 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" event={"ID":"87855fb7-df0f-468b-9b00-55f680e3a242","Type":"ContainerDied","Data":"8cb06e0970f886d97059559171ac1bafdb4c137f2069ca41e6e1e732cf6c417b"} Oct 30 13:21:17.693851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:17.693830 5049 scope.go:122] "RemoveContainer" containerID="0aa25785191760e8c0dc85f46d8d8f26d0eb5d69731190d44934585eecb1c896" Oct 30 13:21:17.694074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:17.693787 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m" Oct 30 13:21:17.841262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:17.841201 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m"] Oct 30 13:21:17.847751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:17.847717 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-d7w7m"] Oct 30 13:21:17.917355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:17.911519 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:19.346925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:19.346892 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" path="/var/lib/kubelet/pods/87855fb7-df0f-468b-9b00-55f680e3a242/volumes" Oct 30 13:21:19.984376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:19.982913 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:21.911923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:21.911844 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:23.478565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:23.478520 5049 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:21:23.479003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:23.478600 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:21:23.768383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:23.768257 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:24.598426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:24.598394 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:25.721757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:25.721719 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:25.811055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:25.811018 5049 generic.go:356] "Generic (PLEG): container finished" podID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerID="795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c" exitCode=0 Oct 30 13:21:25.811226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:25.811094 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" event={"ID":"1a94e7e5-3c0e-4f97-a16c-622e809fcd71","Type":"ContainerDied","Data":"795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c"} Oct 30 13:21:25.811710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:25.811692 5049 scope.go:122] "RemoveContainer" containerID="795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c" Oct 30 13:21:25.814037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:25.814001 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:21:25.814153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:25.814106 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:21:25.814201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:25.814156 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:21:25.890031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:25.889954 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:26.391337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:26.391292 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8"] Oct 30 13:21:26.391572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:26.391538 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" containerID="cri-o://ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080" gracePeriod=30 Oct 30 13:21:26.678798 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:21:26.678714 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:21:26.822619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:26.822576 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" event={"ID":"1a94e7e5-3c0e-4f97-a16c-622e809fcd71","Type":"ContainerStarted","Data":"edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1"} Oct 30 13:21:27.365147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.364779 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:21:27.365147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.364909 5049 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 13:21:27.365147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.364952 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 13:21:27.443007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.442959 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh"] Oct 30 13:21:27.444077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.444022 5049 cpu_manager.go:414] "containerMap: removing container" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" Oct 30 13:21:27.444077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.444046 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" Oct 30 13:21:27.444246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.444118 5049 cpu_manager.go:414] "containerMap: removing container" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" Oct 30 13:21:27.444246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.444127 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" Oct 30 13:21:27.444246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.444149 5049 cpu_manager.go:414] "containerMap: removing container" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" Oct 30 13:21:27.444246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.444158 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" Oct 30 13:21:27.444400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.444356 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" Oct 30 13:21:27.444400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.444374 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" Oct 30 13:21:27.444400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.444385 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" Oct 30 13:21:27.444498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.444402 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="87855fb7-df0f-468b-9b00-55f680e3a242" containerName="istio-proxy" Oct 30 13:21:27.448490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.448471 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.462441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.462415 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh"] Oct 30 13:21:27.467835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.467810 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ddb271-openshift-default-dockercfg-7fnrt\"" Oct 30 13:21:27.507946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.507906 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:27.548188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.548150 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-credential-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.548570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.548544 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-envoy\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.548764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.548746 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.548954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.548936 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istiod-ca-cert\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.549192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.549173 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-token\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.549396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.549377 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-podinfo\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.549599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.549581 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-data\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.549768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.549748 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-certs\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.549977 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.549959 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7f8k\" (UniqueName: \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-kube-api-access-p7f8k\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.661817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.661718 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-envoy\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.662765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.662742 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.662963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.662375 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.663175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.663157 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istiod-ca-cert\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.664349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.664196 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istiod-ca-cert\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.664578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.664564 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-token\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.665088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.665069 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-podinfo\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.665761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.665740 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-data\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.666331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.666150 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-data\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.666472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.666286 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-certs\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.666945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.666847 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-certs\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.670432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.668506 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7f8k\" (UniqueName: \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-kube-api-access-p7f8k\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.670432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.668727 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-credential-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.670432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.669090 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-credential-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.671101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.671080 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-podinfo\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.671445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.671424 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-envoy\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.686667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.686641 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-token\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.703080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.697089 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:27.724343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.705890 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7f8k\" (UniqueName: \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-kube-api-access-p7f8k\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.769771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.768568 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:27.857584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:27.857548 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:28.172408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:28.172373 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh"] Oct 30 13:21:28.176516 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:21:28.176486 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb3d2a9_aadb_48a9_a2d7_4b5b97c7bf0a.slice/crio-affa42afeac4b89529309a2a06da6e7839ceee7148ba247bd8d45b83e7134899 WatchSource:0}: Error finding container affa42afeac4b89529309a2a06da6e7839ceee7148ba247bd8d45b83e7134899: Status 404 returned error can't find the container with id affa42afeac4b89529309a2a06da6e7839ceee7148ba247bd8d45b83e7134899 Oct 30 13:21:28.179460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:28.179430 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:21:28.179554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:28.179496 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:21:28.179554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:28.179528 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:21:28.365197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:28.365159 5049 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 13:21:28.365627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:28.365224 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 13:21:28.872456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:28.872417 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" event={"ID":"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a","Type":"ContainerStarted","Data":"dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40"} Oct 30 13:21:28.872880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:28.872487 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" event={"ID":"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a","Type":"ContainerStarted","Data":"affa42afeac4b89529309a2a06da6e7839ceee7148ba247bd8d45b83e7134899"} Oct 30 13:21:28.958345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:28.954719 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" podStartSLOduration=1.954698854 podStartE2EDuration="1.954698854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:28.926766377 +0000 UTC m=+1795.741960800" watchObservedRunningTime="2026-10-30 13:21:28.954698854 +0000 UTC m=+1795.769893277" Oct 30 13:21:29.376346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:29.371219 5049 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 13:21:29.376346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:29.371281 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 13:21:29.524346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:29.523954 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:29.769404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:29.769198 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:29.771181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:29.771058 5049 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 13:21:29.771181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:29.771110 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 13:21:30.372959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:30.372918 5049 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 13:21:30.373508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:30.372977 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 13:21:30.776599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:30.776514 5049 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 13:21:30.778811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:30.778275 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 13:21:31.356887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.356849 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:31.382849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.382804 5049 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 13:21:31.383481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.383443 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 13:21:31.775207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.773939 5049 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 13:21:31.775207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.773999 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 13:21:31.919396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.917501 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:21:31.952345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.951715 5049 generic.go:356] "Generic (PLEG): container finished" podID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerID="ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080" exitCode=0 Oct 30 13:21:31.952345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.952033 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" event={"ID":"9f2f7fb4-e4bd-49c4-a070-bed24a63733e","Type":"ContainerDied","Data":"ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080"} Oct 30 13:21:31.952345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.952090 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" event={"ID":"9f2f7fb4-e4bd-49c4-a070-bed24a63733e","Type":"ContainerDied","Data":"9f8419d3fa9fb109420ca5f4bf312037f5fe69426e9e560992da54bb0c9220d8"} Oct 30 13:21:31.952345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.952121 5049 scope.go:122] "RemoveContainer" containerID="ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080" Oct 30 13:21:31.977226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.977190 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-podinfo\") pod \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " Oct 30 13:21:31.977456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.977261 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-envoy\") pod \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " Oct 30 13:21:31.977456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.977291 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-socket\") pod \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " Oct 30 13:21:31.977456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.977408 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-certs\") pod \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " Oct 30 13:21:31.977456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.977431 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istiod-ca-cert\") pod \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " Oct 30 13:21:31.977456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.977456 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-token\" (UniqueName: \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-token\") pod \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " Oct 30 13:21:31.977726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.977502 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-kube-api-access-l4mcb\" (UniqueName: \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-kube-api-access-l4mcb\") pod \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " Oct 30 13:21:31.977726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.977586 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-credential-socket\") pod \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " Oct 30 13:21:31.977726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.977673 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-data\") pod \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\" (UID: \"9f2f7fb4-e4bd-49c4-a070-bed24a63733e\") " Oct 30 13:21:31.978287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.978267 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-data" pod "9f2f7fb4-e4bd-49c4-a070-bed24a63733e" (UID: "9f2f7fb4-e4bd-49c4-a070-bed24a63733e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:31.979217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.978939 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-socket" pod "9f2f7fb4-e4bd-49c4-a070-bed24a63733e" (UID: "9f2f7fb4-e4bd-49c4-a070-bed24a63733e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:31.979217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.978887 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istiod-ca-cert" pod "9f2f7fb4-e4bd-49c4-a070-bed24a63733e" (UID: "9f2f7fb4-e4bd-49c4-a070-bed24a63733e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:31.979217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.979184 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-credential-socket" pod "9f2f7fb4-e4bd-49c4-a070-bed24a63733e" (UID: "9f2f7fb4-e4bd-49c4-a070-bed24a63733e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:31.979671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.979645 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-certs" pod "9f2f7fb4-e4bd-49c4-a070-bed24a63733e" (UID: "9f2f7fb4-e4bd-49c4-a070-bed24a63733e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:31.980715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.980691 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-podinfo" pod "9f2f7fb4-e4bd-49c4-a070-bed24a63733e" (UID: "9f2f7fb4-e4bd-49c4-a070-bed24a63733e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:21:31.981411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.981392 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-envoy" pod "9f2f7fb4-e4bd-49c4-a070-bed24a63733e" (UID: "9f2f7fb4-e4bd-49c4-a070-bed24a63733e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:31.981583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.981425 5049 scope.go:122] "RemoveContainer" containerID="ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080" Oct 30 13:21:31.982096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.982069 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-kube-api-access-l4mcb" pod "9f2f7fb4-e4bd-49c4-a070-bed24a63733e" (UID: "9f2f7fb4-e4bd-49c4-a070-bed24a63733e"). InnerVolumeSpecName "kube-api-access-l4mcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:31.982309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.982274 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-token" pod "9f2f7fb4-e4bd-49c4-a070-bed24a63733e" (UID: "9f2f7fb4-e4bd-49c4-a070-bed24a63733e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:31.982455 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:21:31.982306 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080\": container with ID starting with ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080 not found: ID does not exist" containerID="ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080" Oct 30 13:21:31.982455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:31.982364 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080"} err="failed to get container status \"ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080\": rpc error: code = NotFound desc = could not find container \"ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080\": container with ID starting with ce362e2f27387b0373f32dc661da798b302efe4de737012fd0b46e132e9c7080 not found: ID does not exist" Oct 30 13:21:32.079596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.079546 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:32.079596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.079588 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:32.079596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.079599 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:32.079596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.079607 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:32.079981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.079616 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:32.079981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.079623 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:32.079981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.079631 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4mcb\" (UniqueName: \"kubernetes.io/projected/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-kube-api-access-l4mcb\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:32.079981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.079640 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:32.079981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.079650 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2f7fb4-e4bd-49c4-a070-bed24a63733e-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:21:32.367086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.367043 5049 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 13:21:32.367309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.367117 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 13:21:32.769653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.769561 5049 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 13:21:32.769653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.769616 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 13:21:32.960826 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:32.960794 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8" Oct 30 13:21:33.044747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.044718 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8"] Oct 30 13:21:33.051244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.051215 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-wm4q8"] Oct 30 13:21:33.151345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.150898 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:33.348774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.348692 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" path="/var/lib/kubelet/pods/9f2f7fb4-e4bd-49c4-a070-bed24a63733e/volumes" Oct 30 13:21:33.369659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.369630 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:21:33.370088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.370071 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:21:33.370750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.370730 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:21:33.665032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.664957 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:21:33.669657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.669635 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:21:33.769790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.769756 5049 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 13:21:33.770287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:33.769860 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 13:21:34.770063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:34.770012 5049 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 13:21:34.770572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:34.770082 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 13:21:34.873882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:34.873828 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:35.775705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:35.775632 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:35.776135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:35.776111 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:35.776773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:35.776753 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:21:36.739743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:36.739703 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:36.892064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:36.892030 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:38.748650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:38.748609 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:40.513779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:40.513742 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:42.307639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:42.307578 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:44.112893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.112856 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g"] Oct 30 13:21:44.113680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.113663 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" Oct 30 13:21:44.113725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.113680 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" Oct 30 13:21:44.113725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.113708 5049 cpu_manager.go:414] "containerMap: removing container" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" Oct 30 13:21:44.113725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.113714 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e20740-a55b-4ca5-8f06-f01b8b9cd57d" containerName="istio-proxy" Oct 30 13:21:44.113845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.113832 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2f7fb4-e4bd-49c4-a070-bed24a63733e" containerName="istio-proxy" Oct 30 13:21:44.119410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.119393 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.144622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.144578 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-09e4c0-serving-cert\"" Oct 30 13:21:44.150184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.150155 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g"] Oct 30 13:21:44.229817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.229776 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-metrics-certs\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.230069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.230042 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-maas-api-tls\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.230141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.230090 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11b2731-225a-4b4a-9b36-76d4a44e2239-openshift-service-ca\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.230243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.230212 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcd2\" (UniqueName: \"kubernetes.io/projected/e11b2731-225a-4b4a-9b36-76d4a44e2239-kube-api-access-hbcd2\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.250024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.249994 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:44.355345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.335934 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-metrics-certs\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.355967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.355931 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-maas-api-tls\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.356246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.356150 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11b2731-225a-4b4a-9b36-76d4a44e2239-openshift-service-ca\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.369447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.356432 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcd2\" (UniqueName: \"kubernetes.io/projected/e11b2731-225a-4b4a-9b36-76d4a44e2239-kube-api-access-hbcd2\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.375376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.373338 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11b2731-225a-4b4a-9b36-76d4a44e2239-openshift-service-ca\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.380703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.380661 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-metrics-certs\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.381171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.381149 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-maas-api-tls\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.410184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.410153 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcd2\" (UniqueName: \"kubernetes.io/projected/e11b2731-225a-4b4a-9b36-76d4a44e2239-kube-api-access-hbcd2\") pod \"maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.426383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.420622 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:44.434577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.434547 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:44.590171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.590137 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:44.761580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:44.760626 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g"] Oct 30 13:21:44.764390 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:21:44.763947 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11b2731_225a_4b4a_9b36_76d4a44e2239.slice/crio-8f29b564fd88a7e22ab1145784d3f0390b7741a364dfaa27c05acaec04234966 WatchSource:0}: Error finding container 8f29b564fd88a7e22ab1145784d3f0390b7741a364dfaa27c05acaec04234966: Status 404 returned error can't find the container with id 8f29b564fd88a7e22ab1145784d3f0390b7741a364dfaa27c05acaec04234966 Oct 30 13:21:45.119024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:45.118987 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" event={"ID":"e11b2731-225a-4b4a-9b36-76d4a44e2239","Type":"ContainerStarted","Data":"8f29b564fd88a7e22ab1145784d3f0390b7741a364dfaa27c05acaec04234966"} Oct 30 13:21:46.130529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:46.130493 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" event={"ID":"e11b2731-225a-4b4a-9b36-76d4a44e2239","Type":"ContainerStarted","Data":"d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a"} Oct 30 13:21:46.131033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:46.130606 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:46.132474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:46.132425 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:46.179957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:46.179885 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" podStartSLOduration=1.820211514 podStartE2EDuration="2.17986522s" totalImagesPullingTime="359.653706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:46.176123761 +0000 UTC m=+1812.991318189" watchObservedRunningTime="2026-10-30 13:21:46.17986522 +0000 UTC m=+1812.995059644" Oct 30 13:21:46.341183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:46.341120 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:46.600611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:46.600558 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:46.989001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:46.988917 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg"] Oct 30 13:21:46.994557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:46.994532 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.030343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.030298 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg"] Oct 30 13:21:47.071934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.071902 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5c7ba3-openshift-default-dockercfg-62dvk\"" Oct 30 13:21:47.125362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.123621 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-istio-token\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.125362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.123676 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e04f33-e190-41e9-9c5d-2390835d027f-istiod-ca-cert\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.125362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.123778 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-envoy\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.125362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.123853 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-data\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.125362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.123912 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb8vs\" (UniqueName: \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-kube-api-access-fb8vs\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.125362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.124014 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-credential-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.125362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.124050 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.125362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.124225 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-certs\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.125362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.124348 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e04f33-e190-41e9-9c5d-2390835d027f-istio-podinfo\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.235353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.235298 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-credential-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.236091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.236070 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.236681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.236660 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-certs\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.237195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.237175 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e04f33-e190-41e9-9c5d-2390835d027f-istio-podinfo\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.253009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.252931 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-istio-token\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.253566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.253540 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e04f33-e190-41e9-9c5d-2390835d027f-istiod-ca-cert\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.258377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.254567 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-envoy\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.282934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.282814 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-data\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.283425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.283306 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb8vs\" (UniqueName: \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-kube-api-access-fb8vs\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.283877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.254434 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e04f33-e190-41e9-9c5d-2390835d027f-istiod-ca-cert\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.284050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.236487 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.284206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.235985 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-credential-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.284348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.237069 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-certs\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.291725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.291699 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-data\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.312001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.311085 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:47.312001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.311918 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e04f33-e190-41e9-9c5d-2390835d027f-istio-podinfo\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.343341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.329581 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-envoy\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.343341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.330062 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-istio-token\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.368952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.368921 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb8vs\" (UniqueName: \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-kube-api-access-fb8vs\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.571658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.571621 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:47.637166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.637132 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:47.971803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.971776 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg"] Oct 30 13:21:47.974105 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:21:47.974067 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e04f33_e190_41e9_9c5d_2390835d027f.slice/crio-a4c425acba8f5430eea000060e0b7494b6fd5a47be25dd7f8ff7faecb1ce9ceb WatchSource:0}: Error finding container a4c425acba8f5430eea000060e0b7494b6fd5a47be25dd7f8ff7faecb1ce9ceb: Status 404 returned error can't find the container with id a4c425acba8f5430eea000060e0b7494b6fd5a47be25dd7f8ff7faecb1ce9ceb Oct 30 13:21:47.977174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.977141 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:21:47.977281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.977206 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:21:47.977281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:47.977237 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:21:48.232738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:48.232643 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" event={"ID":"43e04f33-e190-41e9-9c5d-2390835d027f","Type":"ContainerStarted","Data":"58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14"} Oct 30 13:21:48.232738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:48.232701 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" event={"ID":"43e04f33-e190-41e9-9c5d-2390835d027f","Type":"ContainerStarted","Data":"a4c425acba8f5430eea000060e0b7494b6fd5a47be25dd7f8ff7faecb1ce9ceb"} Oct 30 13:21:48.279469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:48.279412 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" podStartSLOduration=2.279395511 podStartE2EDuration="2.279395511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:48.27591581 +0000 UTC m=+1815.091110247" watchObservedRunningTime="2026-10-30 13:21:48.279395511 +0000 UTC m=+1815.094589934" Oct 30 13:21:48.367028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:48.366990 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:48.637708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:48.637672 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:49.644381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:49.644311 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:50.254946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:50.254908 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:50.256014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:50.255989 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:21:50.624700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:50.624664 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:50.825207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:50.825173 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:51.019828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:51.019745 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:52.230693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:52.230665 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:21:52.800794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:52.800748 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:52.960947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:52.960779 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:53.134345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:53.134236 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:54.901035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:54.900990 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:56.835568 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:21:56.835514 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:21:57.114064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:57.113983 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:59.194134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:21:59.194083 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:00.893053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:00.893018 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:01.132306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:01.132263 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:01.833343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:01.829236 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:02.401183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:02.401145 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:02.845910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:02.845867 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:03.040562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:03.040521 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:03.541453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:03.540796 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:05.492954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:05.492914 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:07.501011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:07.500978 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:08.662290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:08.662249 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:08.822130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:08.822086 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:09.806119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:09.805963 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:11.239260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:11.239219 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:11.434461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:11.434414 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:11.753226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:11.753191 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:12.034798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:12.034707 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:13.742074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:13.742028 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:14.146731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:14.146635 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:16.017567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.017529 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt"] Oct 30 13:22:16.024835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.024810 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.037659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.037635 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-8d82d5-openshift-default-dockercfg-w5n4b\"" Oct 30 13:22:16.054896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.054865 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt"] Oct 30 13:22:16.068243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.068205 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.068427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.068279 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.068530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.068505 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.068723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.068699 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j87cf\" (UniqueName: \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-kube-api-access-j87cf\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.069004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.068982 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.069512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.069492 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.069770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.069745 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.071090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.069819 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.071203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.071151 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.172563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.172533 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.172739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.172590 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.172739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.172641 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.172739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.172676 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.172932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.172757 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.172932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.172786 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.172932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.172842 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.172932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.172875 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j87cf\" (UniqueName: \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-kube-api-access-j87cf\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.172932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.172907 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.173213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.172954 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.173213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.173005 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.173353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.173306 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.173497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.173473 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.173563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.173543 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.175163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.175145 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.175427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.175412 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.194200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.194171 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.194306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.194237 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j87cf\" (UniqueName: \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-kube-api-access-j87cf\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.342191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.342156 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:16.405667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.405624 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:16.731857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.731830 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt"] Oct 30 13:22:16.734443 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:22:16.734418 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852ca5dd_65ad_4ee2_a3b4_b38203b81454.slice/crio-445c86a1fd1880c46ced2eaba4ceed75cda362b2e6b454c95243dd2931d45346 WatchSource:0}: Error finding container 445c86a1fd1880c46ced2eaba4ceed75cda362b2e6b454c95243dd2931d45346: Status 404 returned error can't find the container with id 445c86a1fd1880c46ced2eaba4ceed75cda362b2e6b454c95243dd2931d45346 Oct 30 13:22:16.738233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.738190 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:16.738349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.738278 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:16.738424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.738362 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:16.752656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:16.752629 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" event={"ID":"852ca5dd-65ad-4ee2-a3b4-b38203b81454","Type":"ContainerStarted","Data":"445c86a1fd1880c46ced2eaba4ceed75cda362b2e6b454c95243dd2931d45346"} Oct 30 13:22:17.763063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:17.763014 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" event={"ID":"852ca5dd-65ad-4ee2-a3b4-b38203b81454","Type":"ContainerStarted","Data":"d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d"} Oct 30 13:22:17.818368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:17.818291 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" podStartSLOduration=2.818277129 podStartE2EDuration="2.818277129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:17.817949608 +0000 UTC m=+1844.633144057" watchObservedRunningTime="2026-10-30 13:22:17.818277129 +0000 UTC m=+1844.633471551" Oct 30 13:22:18.342806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:18.342548 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:18.344734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:18.344696 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 13:22:18.344903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:18.344748 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 13:22:18.366137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:18.366050 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:19.343430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:19.343387 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 13:22:19.343946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:19.343460 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 13:22:20.274184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:20.274147 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:20.349344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:20.343712 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 13:22:20.349344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:20.343791 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 13:22:21.046449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:21.044704 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs"] Oct 30 13:22:21.049057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:21.049029 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5989e3ed-224rs"] Oct 30 13:22:21.343481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:21.343369 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 13:22:21.343481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:21.343431 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 13:22:21.353286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:21.353244 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c126fe-c2ba-43dd-b3da-f420b25146d8" path="/var/lib/kubelet/pods/e8c126fe-c2ba-43dd-b3da-f420b25146d8/volumes" Oct 30 13:22:22.345972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:22.345940 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 13:22:22.346248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:22.346220 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 13:22:22.446906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:22.446873 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:23.343196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:23.343145 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 13:22:23.343493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:23.343218 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 13:22:24.349426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:24.349394 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:24.349925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:24.349898 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:24.350788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:24.350768 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:22:24.392440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:24.392299 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:24.858247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:24.858168 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:25.007031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:25.007001 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:25.165796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:25.165709 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:26.196344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:26.191049 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:26.681929 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:22:26.681895 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:22:27.600216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:27.600182 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:28.008261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:28.008146 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:29.496165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:29.496132 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:29.653409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:29.653374 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:29.816208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:29.816175 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:31.825349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:31.822481 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:32.658249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:32.658221 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:33.859065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:33.859029 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:35.786352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:35.785939 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:37.581344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:37.578028 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:39.365226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:39.363963 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:41.299886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:41.298955 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:41.356541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:41.356423 5049 patch_prober.go:29] interesting pod/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:22:41.356541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:41.356499 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:22:42.046350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:42.046293 5049 generic.go:356] "Generic (PLEG): container finished" podID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerID="29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c" exitCode=0 Oct 30 13:22:42.046545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:42.046340 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" event={"ID":"38e9360b-c35c-47bd-b0fe-1ccad4274ab9","Type":"ContainerDied","Data":"29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c"} Oct 30 13:22:42.046882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:42.046864 5049 scope.go:122] "RemoveContainer" containerID="29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c" Oct 30 13:22:42.049368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:42.049336 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:42.049463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:42.049425 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:42.049502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:42.049472 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:43.057551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:43.057513 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" event={"ID":"38e9360b-c35c-47bd-b0fe-1ccad4274ab9","Type":"ContainerStarted","Data":"eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411"} Oct 30 13:22:43.261342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:43.261245 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:43.431395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:43.429149 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:43.600793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:43.600747 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:43.629341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:43.628742 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:22:43.629341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:43.628880 5049 patch_prober.go:29] interesting pod/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:43.629341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:43.628925 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:22:43.768268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:43.768160 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:44.628348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:44.627711 5049 patch_prober.go:29] interesting pod/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:44.628348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:44.627771 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:22:45.190714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:45.190663 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:45.627730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:45.627694 5049 patch_prober.go:29] interesting pod/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:45.628066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:45.628035 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:22:46.041920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:46.041835 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:46.627345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:46.626700 5049 patch_prober.go:29] interesting pod/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:46.627345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:46.626766 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:22:47.038987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:47.038891 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:47.629799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:47.629758 5049 patch_prober.go:29] interesting pod/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:47.630496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:47.630462 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:22:48.627728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:48.627687 5049 patch_prober.go:29] interesting pod/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:48.627901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:48.627756 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 13:22:48.862476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:48.862441 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:49.631252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:49.631220 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:22:49.631796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:49.631768 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:22:49.632579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:49.632560 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:22:50.912439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:50.912397 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:51.104085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:51.104045 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:52.898158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:52.897942 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:54.628907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:54.623254 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:56.498239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:56.498203 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:56.679294 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:22:56.679253 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:22:58.222782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:58.222753 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:59.916047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:59.916002 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq"] Oct 30 13:22:59.922088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:59.922062 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:22:59.941169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:59.941129 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq"] Oct 30 13:22:59.946005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:59.945972 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-376c89-serving-cert\"" Oct 30 13:22:59.991815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:59.991777 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-maas-api-tls\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:22:59.992028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:59.992001 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rvls\" (UniqueName: \"kubernetes.io/projected/8b67a351-2778-4edb-98ba-fb889a7150a8-kube-api-access-7rvls\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:22:59.992126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:59.992062 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b67a351-2778-4edb-98ba-fb889a7150a8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:22:59.992297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:22:59.992270 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-metrics-certs\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:00.053577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.053545 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:00.094168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.094133 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-metrics-certs\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:00.094381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.094203 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-maas-api-tls\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:00.094381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.094332 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rvls\" (UniqueName: \"kubernetes.io/projected/8b67a351-2778-4edb-98ba-fb889a7150a8-kube-api-access-7rvls\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:00.094381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.094375 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b67a351-2778-4edb-98ba-fb889a7150a8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:00.095071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.095046 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b67a351-2778-4edb-98ba-fb889a7150a8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:00.096703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.096683 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-maas-api-tls\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:00.096792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.096732 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-metrics-certs\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:00.116222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.116187 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rvls\" (UniqueName: \"kubernetes.io/projected/8b67a351-2778-4edb-98ba-fb889a7150a8-kube-api-access-7rvls\") pod \"maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:00.212619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.212487 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:00.240664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.240628 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:00.564630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.564603 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq"] Oct 30 13:23:00.567168 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:23:00.567142 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b67a351_2778_4edb_98ba_fb889a7150a8.slice/crio-39c92ec375ec87b2641911ec5e7c8280c44d90c33ee90c27af2a431849d249b9 WatchSource:0}: Error finding container 39c92ec375ec87b2641911ec5e7c8280c44d90c33ee90c27af2a431849d249b9: Status 404 returned error can't find the container with id 39c92ec375ec87b2641911ec5e7c8280c44d90c33ee90c27af2a431849d249b9 Oct 30 13:23:00.568785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:00.568768 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:23:01.253706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:01.253617 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" event={"ID":"8b67a351-2778-4edb-98ba-fb889a7150a8","Type":"ContainerStarted","Data":"ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9"} Oct 30 13:23:01.254289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:01.254262 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" event={"ID":"8b67a351-2778-4edb-98ba-fb889a7150a8","Type":"ContainerStarted","Data":"39c92ec375ec87b2641911ec5e7c8280c44d90c33ee90c27af2a431849d249b9"} Oct 30 13:23:01.255745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:01.255724 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:01.327352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:01.325112 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" podStartSLOduration=1.997624349 podStartE2EDuration="2.325043598s" totalImagesPullingTime="327.419249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:01.324361861 +0000 UTC m=+1888.139556284" watchObservedRunningTime="2026-10-30 13:23:01.325043598 +0000 UTC m=+1888.140238028" Oct 30 13:23:02.096030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:02.095850 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:03.834035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:03.833995 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:05.522753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:05.522722 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:05.679383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:05.679349 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:07.467166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:07.467132 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:08.276098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:08.276069 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:23:09.190156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:09.190121 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:10.058754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:10.058718 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:11.009583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:11.009548 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:12.944223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:12.944159 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:14.803639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:14.800469 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:16.577387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:16.567254 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:18.380586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:18.380548 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:18.538035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:18.537995 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:20.317524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:20.317455 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:22.333400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:22.329150 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s"] Oct 30 13:23:22.333400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:22.329435 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" containerID="cri-o://bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92" gracePeriod=30 Oct 30 13:23:22.457488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:22.436964 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:22.482519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:22.482394 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq"] Oct 30 13:23:22.659402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:22.483063 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" containerID="cri-o://b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476" gracePeriod=30 Oct 30 13:23:22.768608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:22.768569 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:22.842345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:22.837635 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr"] Oct 30 13:23:22.842345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:22.838038 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" containerID="cri-o://b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3" gracePeriod=30 Oct 30 13:23:24.472512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:24.472472 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:26.275988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:26.275942 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:26.678240 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:23:26.678200 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:23:27.724333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.724225 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:27.789552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.789520 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:23:27.967211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.967178 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-token\" (UniqueName: \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-token\") pod \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " Oct 30 13:23:27.967365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.967302 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-socket\") pod \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " Oct 30 13:23:27.967469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.967420 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istiod-ca-cert\") pod \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " Oct 30 13:23:27.967543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.967480 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-kube-api-access-52lvj\" (UniqueName: \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-kube-api-access-52lvj\") pod \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " Oct 30 13:23:27.967616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.967540 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-certs\") pod \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " Oct 30 13:23:27.967675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.967620 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-envoy\") pod \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " Oct 30 13:23:27.967757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.967739 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-data\") pod \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " Oct 30 13:23:27.967911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.967893 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-podinfo\") pod \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " Oct 30 13:23:27.967987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.967949 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-credential-socket\") pod \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\" (UID: \"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f\") " Oct 30 13:23:27.968703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.968674 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-socket" pod "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" (UID: "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:27.968835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.968815 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:27.969398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.969165 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istiod-ca-cert" pod "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" (UID: "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:27.969565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.969462 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-data" pod "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" (UID: "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:27.969565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.969520 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-certs" pod "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" (UID: "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:27.970170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.970142 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-credential-socket" pod "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" (UID: "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:27.971480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.971453 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-envoy" pod "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" (UID: "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:27.971816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.971769 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-podinfo" pod "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" (UID: "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:23:27.990693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.990587 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-token" pod "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" (UID: "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:27.991620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:27.991503 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-kube-api-access-52lvj" pod "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" (UID: "d8a1172f-16fa-43a9-8c23-bfcd541fbc0f"). InnerVolumeSpecName "kube-api-access-52lvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:28.071353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.071206 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.071353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.071245 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52lvj\" (UniqueName: \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-kube-api-access-52lvj\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.071353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.071261 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.071353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.071279 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.071353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.071292 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.071353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.071305 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.072284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.072205 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.072284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.072239 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.145379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.136995 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:28.211117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.211081 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:23:28.290650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.290551 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-kube-api-access-dqsxb\" (UniqueName: \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-kube-api-access-dqsxb\") pod \"963fea52-5fa2-4b2e-be70-c18f8621f46c\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " Oct 30 13:23:28.290650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.290605 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-data\") pod \"963fea52-5fa2-4b2e-be70-c18f8621f46c\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " Oct 30 13:23:28.290926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.290701 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-envoy\") pod \"963fea52-5fa2-4b2e-be70-c18f8621f46c\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " Oct 30 13:23:28.290926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.290760 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-credential-socket\") pod \"963fea52-5fa2-4b2e-be70-c18f8621f46c\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " Oct 30 13:23:28.290926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.290824 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-certs\") pod \"963fea52-5fa2-4b2e-be70-c18f8621f46c\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " Oct 30 13:23:28.290926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.290864 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-token\" (UniqueName: \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-token\") pod \"963fea52-5fa2-4b2e-be70-c18f8621f46c\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " Oct 30 13:23:28.290926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.290894 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-socket\") pod \"963fea52-5fa2-4b2e-be70-c18f8621f46c\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " Oct 30 13:23:28.291813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.291633 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-credential-socket" pod "963fea52-5fa2-4b2e-be70-c18f8621f46c" (UID: "963fea52-5fa2-4b2e-be70-c18f8621f46c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:28.292165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.291764 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.292165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.291827 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-socket" pod "963fea52-5fa2-4b2e-be70-c18f8621f46c" (UID: "963fea52-5fa2-4b2e-be70-c18f8621f46c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:28.292165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.291986 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-certs" pod "963fea52-5fa2-4b2e-be70-c18f8621f46c" (UID: "963fea52-5fa2-4b2e-be70-c18f8621f46c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:28.292503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.292464 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-data" pod "963fea52-5fa2-4b2e-be70-c18f8621f46c" (UID: "963fea52-5fa2-4b2e-be70-c18f8621f46c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:28.294122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.294099 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-kube-api-access-dqsxb" pod "963fea52-5fa2-4b2e-be70-c18f8621f46c" (UID: "963fea52-5fa2-4b2e-be70-c18f8621f46c"). InnerVolumeSpecName "kube-api-access-dqsxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:28.295891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.295858 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-envoy" pod "963fea52-5fa2-4b2e-be70-c18f8621f46c" (UID: "963fea52-5fa2-4b2e-be70-c18f8621f46c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:28.298821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.298786 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-token" pod "963fea52-5fa2-4b2e-be70-c18f8621f46c" (UID: "963fea52-5fa2-4b2e-be70-c18f8621f46c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:28.358068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.358040 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:23:28.392568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.392535 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-socket\") pod \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " Oct 30 13:23:28.392765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.392635 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-credential-socket\") pod \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " Oct 30 13:23:28.392765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.392681 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-certs\") pod \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " Oct 30 13:23:28.392765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.392732 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-token\" (UniqueName: \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-token\") pod \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " Oct 30 13:23:28.392951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.392774 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-envoy\") pod \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " Oct 30 13:23:28.392951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.392898 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istiod-ca-cert\") pod \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " Oct 30 13:23:28.392951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.392935 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-data\") pod \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " Oct 30 13:23:28.393120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.393080 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-kube-api-access-w4mw7\" (UniqueName: \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-kube-api-access-w4mw7\") pod \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " Oct 30 13:23:28.393179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.393138 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963fea52-5fa2-4b2e-be70-c18f8621f46c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963fea52-5fa2-4b2e-be70-c18f8621f46c-istiod-ca-cert\") pod \"963fea52-5fa2-4b2e-be70-c18f8621f46c\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " Oct 30 13:23:28.393668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.393258 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-podinfo\") pod \"963fea52-5fa2-4b2e-be70-c18f8621f46c\" (UID: \"963fea52-5fa2-4b2e-be70-c18f8621f46c\") " Oct 30 13:23:28.393668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.393301 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-podinfo\") pod \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\" (UID: \"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5\") " Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.393300 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-credential-socket" pod "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" (UID: "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.393354 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istiod-ca-cert" pod "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" (UID: "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.393421 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-certs" pod "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" (UID: "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.393520 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-socket" pod "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" (UID: "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.393639 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963fea52-5fa2-4b2e-be70-c18f8621f46c-istiod-ca-cert" pod "963fea52-5fa2-4b2e-be70-c18f8621f46c" (UID: "963fea52-5fa2-4b2e-be70-c18f8621f46c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.393904 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-data" pod "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" (UID: "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394183 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqsxb\" (UniqueName: \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-kube-api-access-dqsxb\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394203 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394217 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394230 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394246 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394259 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394275 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394287 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394299 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963fea52-5fa2-4b2e-be70-c18f8621f46c-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394314 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394347 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.394441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.394360 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963fea52-5fa2-4b2e-be70-c18f8621f46c-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.397365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.396530 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-podinfo" pod "963fea52-5fa2-4b2e-be70-c18f8621f46c" (UID: "963fea52-5fa2-4b2e-be70-c18f8621f46c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:23:28.397365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.396564 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-podinfo" pod "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" (UID: "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:23:28.397813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.397795 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-envoy" pod "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" (UID: "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:28.397891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.397812 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-token" pod "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" (UID: "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:28.397891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.397852 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-kube-api-access-w4mw7" pod "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" (UID: "3d7ae366-ee2c-4d84-a62b-2e37405fbcf5"). InnerVolumeSpecName "kube-api-access-w4mw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:28.475374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.474409 5049 generic.go:356] "Generic (PLEG): container finished" podID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerID="b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476" exitCode=0 Oct 30 13:23:28.475374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.474613 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" Oct 30 13:23:28.475374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.474649 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" event={"ID":"963fea52-5fa2-4b2e-be70-c18f8621f46c","Type":"ContainerDied","Data":"b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476"} Oct 30 13:23:28.475374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.474709 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq" event={"ID":"963fea52-5fa2-4b2e-be70-c18f8621f46c","Type":"ContainerDied","Data":"92085ff4aa4dc407d58284d3570ae93bf5396dfe762804fca3867936f239d83f"} Oct 30 13:23:28.475374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.474740 5049 scope.go:122] "RemoveContainer" containerID="b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476" Oct 30 13:23:28.489079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.488823 5049 generic.go:356] "Generic (PLEG): container finished" podID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerID="bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92" exitCode=0 Oct 30 13:23:28.489079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.488872 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" Oct 30 13:23:28.489079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.489000 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" event={"ID":"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f","Type":"ContainerDied","Data":"bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92"} Oct 30 13:23:28.489079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.489039 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s" event={"ID":"d8a1172f-16fa-43a9-8c23-bfcd541fbc0f","Type":"ContainerDied","Data":"62522da1222d908b34cf6617a354a1ea1ef7d513bcffaa72a744dbe8b151f336"} Oct 30 13:23:28.491637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.491610 5049 generic.go:356] "Generic (PLEG): container finished" podID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerID="b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3" exitCode=0 Oct 30 13:23:28.491790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.491773 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" event={"ID":"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5","Type":"ContainerDied","Data":"b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3"} Oct 30 13:23:28.491886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.491819 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" event={"ID":"3d7ae366-ee2c-4d84-a62b-2e37405fbcf5","Type":"ContainerDied","Data":"29d830b31d00088404d7478d705a0641025a77039b8137191273ef7b7083722f"} Oct 30 13:23:28.492034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.491888 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr" Oct 30 13:23:28.495745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.495719 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963fea52-5fa2-4b2e-be70-c18f8621f46c-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.495852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.495747 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.495852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.495762 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.495852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.495775 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.495852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.495791 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4mw7\" (UniqueName: \"kubernetes.io/projected/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5-kube-api-access-w4mw7\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:28.502314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.502295 5049 scope.go:122] "RemoveContainer" containerID="b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476" Oct 30 13:23:28.503858 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:23:28.503678 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476\": container with ID starting with b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476 not found: ID does not exist" containerID="b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476" Oct 30 13:23:28.503858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.503717 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476"} err="failed to get container status \"b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476\": rpc error: code = NotFound desc = could not find container \"b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476\": container with ID starting with b9f509e15c06ac630dc69b089fd4e158a06e527c5ae9183c5085cf45b6e67476 not found: ID does not exist" Oct 30 13:23:28.503858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.503742 5049 scope.go:122] "RemoveContainer" containerID="bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92" Oct 30 13:23:28.552385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.548282 5049 scope.go:122] "RemoveContainer" containerID="bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92" Oct 30 13:23:28.572301 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:23:28.557441 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92\": container with ID starting with bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92 not found: ID does not exist" containerID="bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92" Oct 30 13:23:28.572301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.557499 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92"} err="failed to get container status \"bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92\": rpc error: code = NotFound desc = could not find container \"bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92\": container with ID starting with bd7fc028d461f8807fd72eacc65f6019e57d78d5ab9c61b65669835591de8f92 not found: ID does not exist" Oct 30 13:23:28.572301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.557527 5049 scope.go:122] "RemoveContainer" containerID="b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3" Oct 30 13:23:28.577367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.577337 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq"] Oct 30 13:23:28.581575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.581549 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-lzglq"] Oct 30 13:23:28.583031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.583009 5049 scope.go:122] "RemoveContainer" containerID="b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3" Oct 30 13:23:28.583361 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:23:28.583334 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3\": container with ID starting with b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3 not found: ID does not exist" containerID="b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3" Oct 30 13:23:28.583465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.583359 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3"} err="failed to get container status \"b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3\": rpc error: code = NotFound desc = could not find container \"b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3\": container with ID starting with b430e6c31b5aeb242df3da7dccf459ad3f930b5376f77ddf969b73b0f0adb7a3 not found: ID does not exist" Oct 30 13:23:28.599969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.599937 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr"] Oct 30 13:23:28.607139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.607102 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-4zhvr"] Oct 30 13:23:28.639634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.639600 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s"] Oct 30 13:23:28.645951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:28.645923 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-74k2s"] Oct 30 13:23:29.343582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:29.343541 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" path="/var/lib/kubelet/pods/3d7ae366-ee2c-4d84-a62b-2e37405fbcf5/volumes" Oct 30 13:23:29.344357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:29.344313 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" path="/var/lib/kubelet/pods/963fea52-5fa2-4b2e-be70-c18f8621f46c/volumes" Oct 30 13:23:29.345011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:29.344989 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" path="/var/lib/kubelet/pods/d8a1172f-16fa-43a9-8c23-bfcd541fbc0f/volumes" Oct 30 13:23:30.303540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:30.303445 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:32.159674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:32.159579 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:32.539428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:32.539336 5049 generic.go:356] "Generic (PLEG): container finished" podID="43e04f33-e190-41e9-9c5d-2390835d027f" containerID="58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14" exitCode=0 Oct 30 13:23:32.539428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:32.539353 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" event={"ID":"43e04f33-e190-41e9-9c5d-2390835d027f","Type":"ContainerDied","Data":"58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14"} Oct 30 13:23:32.539939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:32.539921 5049 scope.go:122] "RemoveContainer" containerID="58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14" Oct 30 13:23:32.542560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:32.542524 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:23:32.542699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:32.542616 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:23:32.542699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:32.542666 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:23:33.552605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:33.552567 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" event={"ID":"43e04f33-e190-41e9-9c5d-2390835d027f","Type":"ContainerStarted","Data":"bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a"} Oct 30 13:23:33.640283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:33.639711 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:23:33.642457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:33.642425 5049 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 13:23:33.642623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:33.642489 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 13:23:33.882855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:33.882775 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:34.638370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:34.637975 5049 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 13:23:34.638370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:34.638049 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 13:23:35.255703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:35.255669 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:23:35.454599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:35.454566 5049 scope.go:122] "RemoveContainer" containerID="4622f88121f1a7df6a25484dd7d939ed0e27bea8ca396032aa331aca040fe652" Oct 30 13:23:35.466473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:35.466444 5049 scope.go:122] "RemoveContainer" containerID="6374648881ec58c043871a8afd0af3200acf04f5c6272cec9f5a0ced65992cd3" Oct 30 13:23:35.476496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:35.476475 5049 scope.go:122] "RemoveContainer" containerID="5ce4bdc404135658e724dab5f8b5c3d28a89fca7e9288aa0ea387c727115affe" Oct 30 13:23:35.566788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:35.566742 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:35.637856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:35.637814 5049 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 13:23:35.638046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:35.637883 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 13:23:35.746929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:35.746886 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:35.927343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:35.926624 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:36.637972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:36.637930 5049 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 13:23:36.638171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:36.637997 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 13:23:37.659514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:37.659468 5049 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 13:23:37.660026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:37.659560 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 13:23:37.675809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:37.675764 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:38.279306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.279274 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:38.646116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.646064 5049 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 13:23:38.646440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.646397 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 13:23:38.649865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.649842 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll"] Oct 30 13:23:38.651349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.651309 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" Oct 30 13:23:38.651490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.651474 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" Oct 30 13:23:38.651622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.651609 5049 cpu_manager.go:414] "containerMap: removing container" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" Oct 30 13:23:38.651714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.651703 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" Oct 30 13:23:38.651870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.651858 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" Oct 30 13:23:38.651966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.651955 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" Oct 30 13:23:38.652257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.652239 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a1172f-16fa-43a9-8c23-bfcd541fbc0f" containerName="istio-proxy" Oct 30 13:23:38.652401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.652388 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7ae366-ee2c-4d84-a62b-2e37405fbcf5" containerName="istio-proxy" Oct 30 13:23:38.652505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.652493 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="963fea52-5fa2-4b2e-be70-c18f8621f46c" containerName="istio-proxy" Oct 30 13:23:38.666451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.666427 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll"] Oct 30 13:23:38.667041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.667020 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.680836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.680798 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"auth-test-model-8d82d5-kserve-self-signed-certs\"" Oct 30 13:23:38.680988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.680890 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"openshift-service-ca.crt\"" Oct 30 13:23:38.680988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.680962 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"default-dockercfg-792k5\"" Oct 30 13:23:38.690699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.690674 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"kube-root-ca.crt\"" Oct 30 13:23:38.730345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.729796 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:38.813347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.812518 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4489cd39-a475-496d-89f3-f38ddf93b025-tls-certs\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.813347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.812736 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-model-cache\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.813347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.812765 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-tmp-dir\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.813347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.812850 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkbr\" (UniqueName: \"kubernetes.io/projected/4489cd39-a475-496d-89f3-f38ddf93b025-kube-api-access-zgkbr\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.813347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.812952 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-dshm\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.813347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.813004 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-home\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.920557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.916340 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4489cd39-a475-496d-89f3-f38ddf93b025-tls-certs\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.920557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.916538 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-model-cache\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.920557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.916566 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-tmp-dir\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.920557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.916646 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkbr\" (UniqueName: \"kubernetes.io/projected/4489cd39-a475-496d-89f3-f38ddf93b025-kube-api-access-zgkbr\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.920557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.916731 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-dshm\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.920557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.916792 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-home\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.920557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.917258 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-home\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.920557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.918587 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-model-cache\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.932344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.929240 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4489cd39-a475-496d-89f3-f38ddf93b025-tls-certs\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.936342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.933660 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-tmp-dir\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.983388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.978009 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-dshm\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.983388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.982000 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkbr\" (UniqueName: \"kubernetes.io/projected/4489cd39-a475-496d-89f3-f38ddf93b025-kube-api-access-zgkbr\") pod \"auth-test-model-8d82d5-kserve-5b99478c65-d68ll\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:38.991341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:38.989663 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:39.350030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:39.349991 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll"] Oct 30 13:23:39.643235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:39.643145 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:23:39.644473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:39.644449 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:23:39.661890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:39.661856 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" event={"ID":"4489cd39-a475-496d-89f3-f38ddf93b025","Type":"ContainerStarted","Data":"6211ce066ea2fa51ec816e418452ac4fa796e3394159f89647467caa655cdfa1"} Oct 30 13:23:39.662088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:39.661904 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" event={"ID":"4489cd39-a475-496d-89f3-f38ddf93b025","Type":"ContainerStarted","Data":"59eb34d247d2bcd66e9579e56006d6ac937618b0a2d5bb18f628777c06b2e133"} Oct 30 13:23:39.726239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:39.726192 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" podStartSLOduration=1.726178341 podStartE2EDuration="1.726178341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:39.721428035 +0000 UTC m=+1926.536622471" watchObservedRunningTime="2026-10-30 13:23:39.726178341 +0000 UTC m=+1926.541372764" Oct 30 13:23:39.793041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:39.792976 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:39.990658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:39.990568 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:41.607382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:41.606983 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:43.588025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:43.587956 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:45.427315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:45.426357 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:47.190604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:47.185341 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:48.990538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:48.990501 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:49.003457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.003431 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:49.137390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.136617 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:49.597122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.597093 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9"] Oct 30 13:23:49.602102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.602083 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:49.630563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.630532 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9"] Oct 30 13:23:49.634829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.634803 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:23:49.634978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.634959 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:23:49.707534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.707493 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee526181-gxxh9\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:49.707750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.707733 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkspt\" (UniqueName: \"kubernetes.io/projected/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-kube-api-access-pkspt\") pod \"maas-api-revoke-sub-ee526181-gxxh9\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:49.707817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.707770 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee526181-gxxh9\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:49.757693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.757669 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:23:49.809243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.809205 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee526181-gxxh9\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:49.809457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.809421 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkspt\" (UniqueName: \"kubernetes.io/projected/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-kube-api-access-pkspt\") pod \"maas-api-revoke-sub-ee526181-gxxh9\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:49.809535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.809454 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee526181-gxxh9\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:49.810031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.810000 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee526181-gxxh9\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:49.812065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.812036 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee526181-gxxh9\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:49.839363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.832173 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkspt\" (UniqueName: \"kubernetes.io/projected/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-kube-api-access-pkspt\") pod \"maas-api-revoke-sub-ee526181-gxxh9\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:49.918654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:49.918567 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:50.238298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:50.238271 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9"] Oct 30 13:23:50.240251 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:23:50.240219 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abc99ff_a7fb_4801_b89b_e4ba0db7b480.slice/crio-754a38d090441740fec49a7967ec08aa8dffbb61d5c6bf692fc30a35f66f0ce9 WatchSource:0}: Error finding container 754a38d090441740fec49a7967ec08aa8dffbb61d5c6bf692fc30a35f66f0ce9: Status 404 returned error can't find the container with id 754a38d090441740fec49a7967ec08aa8dffbb61d5c6bf692fc30a35f66f0ce9 Oct 30 13:23:50.757095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:50.757059 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" event={"ID":"6abc99ff-a7fb-4801-b89b-e4ba0db7b480","Type":"ContainerStarted","Data":"f46a367166175e3dfb700b594ccb38f663b414d43389eccb1f5c8e342931c2c5"} Oct 30 13:23:50.757298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:50.757114 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" event={"ID":"6abc99ff-a7fb-4801-b89b-e4ba0db7b480","Type":"ContainerStarted","Data":"754a38d090441740fec49a7967ec08aa8dffbb61d5c6bf692fc30a35f66f0ce9"} Oct 30 13:23:50.801195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:50.801134 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" podStartSLOduration=1.801120932 podStartE2EDuration="1.801120932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:50.799292532 +0000 UTC m=+1937.614486956" watchObservedRunningTime="2026-10-30 13:23:50.801120932 +0000 UTC m=+1937.616315371" Oct 30 13:23:51.020992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:51.020900 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:51.767876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:51.767844 5049 generic.go:356] "Generic (PLEG): container finished" podID="6abc99ff-a7fb-4801-b89b-e4ba0db7b480" containerID="f46a367166175e3dfb700b594ccb38f663b414d43389eccb1f5c8e342931c2c5" exitCode=0 Oct 30 13:23:51.768305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:51.767921 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" event={"ID":"6abc99ff-a7fb-4801-b89b-e4ba0db7b480","Type":"ContainerDied","Data":"f46a367166175e3dfb700b594ccb38f663b414d43389eccb1f5c8e342931c2c5"} Oct 30 13:23:52.785632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:52.785596 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:53.010330 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.010296 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:53.065017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.064976 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-xks-ca\") pod \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " Oct 30 13:23:53.065260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.065236 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-openshift-ca\") pod \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " Oct 30 13:23:53.065478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.065456 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-kube-api-access-pkspt\" (UniqueName: \"kubernetes.io/projected/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-kube-api-access-pkspt\") pod \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\" (UID: \"6abc99ff-a7fb-4801-b89b-e4ba0db7b480\") " Oct 30 13:23:53.065692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.065663 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-openshift-ca" pod "6abc99ff-a7fb-4801-b89b-e4ba0db7b480" (UID: "6abc99ff-a7fb-4801-b89b-e4ba0db7b480"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:53.066141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.066110 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:53.068532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.068503 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-xks-ca" pod "6abc99ff-a7fb-4801-b89b-e4ba0db7b480" (UID: "6abc99ff-a7fb-4801-b89b-e4ba0db7b480"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:53.071810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.071782 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-kube-api-access-pkspt" pod "6abc99ff-a7fb-4801-b89b-e4ba0db7b480" (UID: "6abc99ff-a7fb-4801-b89b-e4ba0db7b480"). InnerVolumeSpecName "kube-api-access-pkspt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:53.168073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.168035 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkspt\" (UniqueName: \"kubernetes.io/projected/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-kube-api-access-pkspt\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:53.168073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.168070 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6abc99ff-a7fb-4801-b89b-e4ba0db7b480-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:23:53.835449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.835414 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" event={"ID":"6abc99ff-a7fb-4801-b89b-e4ba0db7b480","Type":"ContainerDied","Data":"754a38d090441740fec49a7967ec08aa8dffbb61d5c6bf692fc30a35f66f0ce9"} Oct 30 13:23:53.835449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.835450 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9" Oct 30 13:23:53.835892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:53.835469 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="754a38d090441740fec49a7967ec08aa8dffbb61d5c6bf692fc30a35f66f0ce9" Oct 30 13:23:54.690369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:54.689670 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:56.044378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:56.043702 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:56.565965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:56.565929 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:56.680865 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:23:56.680829 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:23:58.479270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:23:58.476018 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:00.304412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:00.304374 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:01.803880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:01.803842 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:02.463777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.463740 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l"] Oct 30 13:24:02.464585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.464567 5049 cpu_manager.go:414] "containerMap: removing container" podUID="6abc99ff-a7fb-4801-b89b-e4ba0db7b480" containerName="revoke-keys" Oct 30 13:24:02.464642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.464586 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abc99ff-a7fb-4801-b89b-e4ba0db7b480" containerName="revoke-keys" Oct 30 13:24:02.464755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.464746 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abc99ff-a7fb-4801-b89b-e4ba0db7b480" containerName="revoke-keys" Oct 30 13:24:02.469144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.469124 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:02.504517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.504480 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:24:02.504947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.504925 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:24:02.506258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.506233 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:02.512646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.510606 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l"] Oct 30 13:24:02.522630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.522590 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-240e9cfd-tcw8l\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:02.522923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.522687 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-240e9cfd-tcw8l\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:02.522923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.522784 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7s2\" (UniqueName: \"kubernetes.io/projected/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-kube-api-access-7c7s2\") pod \"maas-api-revoke-sub-240e9cfd-tcw8l\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:02.624510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.624476 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-240e9cfd-tcw8l\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:02.624724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.624557 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-240e9cfd-tcw8l\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:02.624724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.624618 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7s2\" (UniqueName: \"kubernetes.io/projected/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-kube-api-access-7c7s2\") pod \"maas-api-revoke-sub-240e9cfd-tcw8l\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:02.625202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.625164 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-240e9cfd-tcw8l\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:02.627207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.627178 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-240e9cfd-tcw8l\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:02.646451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.646423 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7s2\" (UniqueName: \"kubernetes.io/projected/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-kube-api-access-7c7s2\") pod \"maas-api-revoke-sub-240e9cfd-tcw8l\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:02.812951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:02.812916 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:03.135398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:03.135356 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l"] Oct 30 13:24:03.139302 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:24:03.139271 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958d0bfa_33b3_446b_84d1_9d8d34eb72b1.slice/crio-0b6aae40a6232462319e9459303a8f6c4a913badc75ce4269940fdc88d1f95be WatchSource:0}: Error finding container 0b6aae40a6232462319e9459303a8f6c4a913badc75ce4269940fdc88d1f95be: Status 404 returned error can't find the container with id 0b6aae40a6232462319e9459303a8f6c4a913badc75ce4269940fdc88d1f95be Oct 30 13:24:03.937742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:03.937702 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" event={"ID":"958d0bfa-33b3-446b-84d1-9d8d34eb72b1","Type":"ContainerStarted","Data":"32543271073ea644a851e7ffd79e41d2adcfadda8b1f44da2d6404bc0e684116"} Oct 30 13:24:03.938251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:03.937761 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" event={"ID":"958d0bfa-33b3-446b-84d1-9d8d34eb72b1","Type":"ContainerStarted","Data":"0b6aae40a6232462319e9459303a8f6c4a913badc75ce4269940fdc88d1f95be"} Oct 30 13:24:03.980308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:03.980253 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" podStartSLOduration=1.9802394030000001 podStartE2EDuration="1.980239403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:03.978836317 +0000 UTC m=+1950.794030740" watchObservedRunningTime="2026-10-30 13:24:03.980239403 +0000 UTC m=+1950.795433827" Oct 30 13:24:04.454375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:04.453794 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:04.948699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:04.948663 5049 generic.go:356] "Generic (PLEG): container finished" podID="958d0bfa-33b3-446b-84d1-9d8d34eb72b1" containerID="32543271073ea644a851e7ffd79e41d2adcfadda8b1f44da2d6404bc0e684116" exitCode=0 Oct 30 13:24:04.949215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:04.948749 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" event={"ID":"958d0bfa-33b3-446b-84d1-9d8d34eb72b1","Type":"ContainerDied","Data":"32543271073ea644a851e7ffd79e41d2adcfadda8b1f44da2d6404bc0e684116"} Oct 30 13:24:05.221584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:05.221446 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:06.173523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.173396 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:06.195388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.194958 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:06.279069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.278975 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-kube-api-access-7c7s2\" (UniqueName: \"kubernetes.io/projected/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-kube-api-access-7c7s2\") pod \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " Oct 30 13:24:06.279223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.279186 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-xks-ca\") pod \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " Oct 30 13:24:06.279266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.279245 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-openshift-ca\") pod \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\" (UID: \"958d0bfa-33b3-446b-84d1-9d8d34eb72b1\") " Oct 30 13:24:06.279849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.279820 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-openshift-ca" pod "958d0bfa-33b3-446b-84d1-9d8d34eb72b1" (UID: "958d0bfa-33b3-446b-84d1-9d8d34eb72b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:06.280083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.280062 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:06.281545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.281514 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-xks-ca" pod "958d0bfa-33b3-446b-84d1-9d8d34eb72b1" (UID: "958d0bfa-33b3-446b-84d1-9d8d34eb72b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:06.281887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.281862 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-kube-api-access-7c7s2" pod "958d0bfa-33b3-446b-84d1-9d8d34eb72b1" (UID: "958d0bfa-33b3-446b-84d1-9d8d34eb72b1"). InnerVolumeSpecName "kube-api-access-7c7s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:06.381863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.381816 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c7s2\" (UniqueName: \"kubernetes.io/projected/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-kube-api-access-7c7s2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:06.381863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.381856 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958d0bfa-33b3-446b-84d1-9d8d34eb72b1-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:06.968824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.968797 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" Oct 30 13:24:06.969008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.968835 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l" event={"ID":"958d0bfa-33b3-446b-84d1-9d8d34eb72b1","Type":"ContainerDied","Data":"0b6aae40a6232462319e9459303a8f6c4a913badc75ce4269940fdc88d1f95be"} Oct 30 13:24:06.969008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:06.968881 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b6aae40a6232462319e9459303a8f6c4a913badc75ce4269940fdc88d1f95be" Oct 30 13:24:08.104394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:08.104310 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:09.868290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:09.868244 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:11.040230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:11.040186 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6"] Oct 30 13:24:11.050426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:11.050382 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb670967-ng6z6"] Oct 30 13:24:11.341252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:11.341216 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe" path="/var/lib/kubelet/pods/3ebc8a98-0b2f-4952-90f9-fd71b8c17cfe/volumes" Oct 30 13:24:11.592119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:11.592017 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:11.677818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:11.677782 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf"] Oct 30 13:24:11.678020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:11.677980 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" containerID="cri-o://20cf340d2aa391cf64324c79cba7eb6274ed53ccd291feebd7360bf5e1f3d428" gracePeriod=30 Oct 30 13:24:11.782412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:11.782370 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:12.630338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:12.630291 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:12.780798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:12.780762 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:13.664345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:13.663862 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:15.487958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:15.487921 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:17.101165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.101133 5049 generic.go:356] "Generic (PLEG): container finished" podID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerID="20cf340d2aa391cf64324c79cba7eb6274ed53ccd291feebd7360bf5e1f3d428" exitCode=0 Oct 30 13:24:17.101905 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.101877 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" event={"ID":"dcae7e80-2c37-47f7-8540-791bfbf7011d","Type":"ContainerDied","Data":"20cf340d2aa391cf64324c79cba7eb6274ed53ccd291feebd7360bf5e1f3d428"} Oct 30 13:24:17.377216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.376994 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:17.398344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.393663 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:24:17.502343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.502237 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-token\" (UniqueName: \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-token\") pod \"dcae7e80-2c37-47f7-8540-791bfbf7011d\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " Oct 30 13:24:17.502343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.502349 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcae7e80-2c37-47f7-8540-791bfbf7011d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae7e80-2c37-47f7-8540-791bfbf7011d-istiod-ca-cert\") pod \"dcae7e80-2c37-47f7-8540-791bfbf7011d\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " Oct 30 13:24:17.502642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.502481 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-certs\") pod \"dcae7e80-2c37-47f7-8540-791bfbf7011d\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " Oct 30 13:24:17.502642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.502552 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-podinfo\") pod \"dcae7e80-2c37-47f7-8540-791bfbf7011d\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " Oct 30 13:24:17.502642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.502590 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-credential-socket\") pod \"dcae7e80-2c37-47f7-8540-791bfbf7011d\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " Oct 30 13:24:17.502809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.502657 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-envoy\") pod \"dcae7e80-2c37-47f7-8540-791bfbf7011d\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " Oct 30 13:24:17.502869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.502851 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-kube-api-access-gxftp\" (UniqueName: \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-kube-api-access-gxftp\") pod \"dcae7e80-2c37-47f7-8540-791bfbf7011d\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " Oct 30 13:24:17.502923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.502897 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-socket\") pod \"dcae7e80-2c37-47f7-8540-791bfbf7011d\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " Oct 30 13:24:17.502980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.502971 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-data\") pod \"dcae7e80-2c37-47f7-8540-791bfbf7011d\" (UID: \"dcae7e80-2c37-47f7-8540-791bfbf7011d\") " Oct 30 13:24:17.524365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.523875 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-envoy" pod "dcae7e80-2c37-47f7-8540-791bfbf7011d" (UID: "dcae7e80-2c37-47f7-8540-791bfbf7011d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:17.524365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.524068 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-credential-socket" pod "dcae7e80-2c37-47f7-8540-791bfbf7011d" (UID: "dcae7e80-2c37-47f7-8540-791bfbf7011d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:17.524365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.524292 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcae7e80-2c37-47f7-8540-791bfbf7011d-istiod-ca-cert" pod "dcae7e80-2c37-47f7-8540-791bfbf7011d" (UID: "dcae7e80-2c37-47f7-8540-791bfbf7011d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:17.524675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.524405 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-data" pod "dcae7e80-2c37-47f7-8540-791bfbf7011d" (UID: "dcae7e80-2c37-47f7-8540-791bfbf7011d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:17.539342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.535898 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-token" pod "dcae7e80-2c37-47f7-8540-791bfbf7011d" (UID: "dcae7e80-2c37-47f7-8540-791bfbf7011d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:17.549717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.549507 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-kube-api-access-gxftp" pod "dcae7e80-2c37-47f7-8540-791bfbf7011d" (UID: "dcae7e80-2c37-47f7-8540-791bfbf7011d"). InnerVolumeSpecName "kube-api-access-gxftp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:17.549717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.549590 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-podinfo" pod "dcae7e80-2c37-47f7-8540-791bfbf7011d" (UID: "dcae7e80-2c37-47f7-8540-791bfbf7011d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:24:17.549717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.549605 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-socket" pod "dcae7e80-2c37-47f7-8540-791bfbf7011d" (UID: "dcae7e80-2c37-47f7-8540-791bfbf7011d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:17.549717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.549671 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-certs" pod "dcae7e80-2c37-47f7-8540-791bfbf7011d" (UID: "dcae7e80-2c37-47f7-8540-791bfbf7011d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:17.633192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.633049 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:17.633555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.633533 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxftp\" (UniqueName: \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-kube-api-access-gxftp\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:17.633671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.633656 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:17.633764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.633750 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:17.633850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.633840 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:17.633936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.633926 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae7e80-2c37-47f7-8540-791bfbf7011d-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:17.634026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.634016 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:17.634110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.634100 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae7e80-2c37-47f7-8540-791bfbf7011d-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:17.634200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.634189 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae7e80-2c37-47f7-8540-791bfbf7011d-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:17.761263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:17.761005 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:18.123597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:18.123555 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" event={"ID":"dcae7e80-2c37-47f7-8540-791bfbf7011d","Type":"ContainerDied","Data":"f11def5fc3e181b708e519a92b74b1bfa3fb36e0368927d2d166629f64984c5e"} Oct 30 13:24:18.124206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:18.124182 5049 scope.go:122] "RemoveContainer" containerID="20cf340d2aa391cf64324c79cba7eb6274ed53ccd291feebd7360bf5e1f3d428" Oct 30 13:24:18.124579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:18.124558 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" Oct 30 13:24:18.176275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:18.176236 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf"] Oct 30 13:24:18.186124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:18.186093 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf"] Oct 30 13:24:18.251471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:18.251429 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:24:18.251674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:18.251506 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-6wfkf" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:24:19.338057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:19.338022 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" path="/var/lib/kubelet/pods/dcae7e80-2c37-47f7-8540-791bfbf7011d/volumes" Oct 30 13:24:19.902173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:19.902081 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:21.726594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:21.726480 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:22.246526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:22.246497 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:23.578819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:23.578775 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:25.605514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:25.603165 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:26.678589 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:24:26.678555 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:24:27.334485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:27.334444 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:27.848940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:27.848903 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:29.156894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:29.156858 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:30.940362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:30.940046 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:32.774840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:32.774802 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:33.149859 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:33.149825 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:34.708752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:34.708656 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:35.563561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:35.563522 5049 scope.go:122] "RemoveContainer" containerID="3aa02ddc7e0bf6e2aba6d151cf50da5bca28121ed313c9458c15c0440fcdccae" Oct 30 13:24:35.574672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:35.574652 5049 scope.go:122] "RemoveContainer" containerID="7c331afe792b40d447ffd8e5c49e2bf3e9153d8b5efc8d1f0d658d86ddf2cc3c" Oct 30 13:24:35.586894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:35.586871 5049 scope.go:122] "RemoveContainer" containerID="6006244c9b03bfe5874023d03b3f1698bbd703bd9e140c8e64d18f840b54e1e8" Oct 30 13:24:35.957500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:35.957416 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:36.490180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:36.490132 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:38.314470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:38.314404 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:40.167100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:40.167065 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:41.421238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.421207 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr"] Oct 30 13:24:41.422155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.422136 5049 cpu_manager.go:414] "containerMap: removing container" podUID="958d0bfa-33b3-446b-84d1-9d8d34eb72b1" containerName="revoke-keys" Oct 30 13:24:41.422155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.422156 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958d0bfa-33b3-446b-84d1-9d8d34eb72b1" containerName="revoke-keys" Oct 30 13:24:41.422283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.422169 5049 cpu_manager.go:414] "containerMap: removing container" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" Oct 30 13:24:41.422283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.422174 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" Oct 30 13:24:41.422399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.422311 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="958d0bfa-33b3-446b-84d1-9d8d34eb72b1" containerName="revoke-keys" Oct 30 13:24:41.422399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.422333 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcae7e80-2c37-47f7-8540-791bfbf7011d" containerName="istio-proxy" Oct 30 13:24:41.427723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.427705 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.453754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.453714 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-376c89\"/\"kube-root-ca.crt\"" Oct 30 13:24:41.453985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.453713 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-376c89\"/\"openshift-service-ca.crt\"" Oct 30 13:24:41.464046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.464010 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-376c89\"/\"default-dockercfg-tjprs\"" Oct 30 13:24:41.464046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.464039 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-376c89\"/\"test-model-376c89-kserve-self-signed-certs\"" Oct 30 13:24:41.468528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.468506 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr"] Oct 30 13:24:41.517291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.517259 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:41.563809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.563771 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-model-cache\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.563995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.563893 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqm7w\" (UniqueName: \"kubernetes.io/projected/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-kube-api-access-rqm7w\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.564052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.563993 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-dshm\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.564090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.564048 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tls-certs\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.564090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.564086 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tmp-dir\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.564208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.564196 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-home\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.677233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.677129 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-home\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.677233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.677233 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-model-cache\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.677693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.677538 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqm7w\" (UniqueName: \"kubernetes.io/projected/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-kube-api-access-rqm7w\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.677693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.677676 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-dshm\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.677829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.677714 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tls-certs\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.677829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.677771 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tmp-dir\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.679910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.678368 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tmp-dir\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.679910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.678634 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-home\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.679910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.678852 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-model-cache\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.692339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.684880 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tls-certs\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.714054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.710353 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqm7w\" (UniqueName: \"kubernetes.io/projected/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-kube-api-access-rqm7w\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.723340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.716801 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-dshm\") pod \"test-model-376c89-kserve-7cdc64f4cf-nqgrr\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:41.748918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:41.747435 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:42.145351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:42.133820 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr"] Oct 30 13:24:42.145351 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:24:42.135185 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bf0a3d8_f0b5_4855_a9b9_a552f61415b1.slice/crio-ade23581de8a58654d1fb8e895c6f0bfbd0944a08b598502467cc66497a7a5b8 WatchSource:0}: Error finding container ade23581de8a58654d1fb8e895c6f0bfbd0944a08b598502467cc66497a7a5b8: Status 404 returned error can't find the container with id ade23581de8a58654d1fb8e895c6f0bfbd0944a08b598502467cc66497a7a5b8 Oct 30 13:24:42.211463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:42.210963 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:42.374256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:42.374213 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" event={"ID":"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1","Type":"ContainerStarted","Data":"4eec222047812262ef9441951c85b1782d491b8d034aa394fe758082d81e6536"} Oct 30 13:24:42.374474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:42.374267 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" event={"ID":"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1","Type":"ContainerStarted","Data":"ade23581de8a58654d1fb8e895c6f0bfbd0944a08b598502467cc66497a7a5b8"} Oct 30 13:24:42.414590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:42.414481 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" podStartSLOduration=1.414468559 podStartE2EDuration="1.414468559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:42.414145257 +0000 UTC m=+1989.229339681" watchObservedRunningTime="2026-10-30 13:24:42.414468559 +0000 UTC m=+1989.229662981" Oct 30 13:24:42.752528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:42.752420 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:43.326824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.326788 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk"] Oct 30 13:24:43.335660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.335633 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.355241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.354508 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk"] Oct 30 13:24:43.358775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.358726 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:24:43.358931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.358809 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:24:43.505501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.505459 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.505716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.505639 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.505985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.505959 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrxpr\" (UniqueName: \"kubernetes.io/projected/2b665100-4126-4836-af2c-a48494cfa6a0-kube-api-access-qrxpr\") pod \"maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.607658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.607571 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.607837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.607661 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.607837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.607796 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrxpr\" (UniqueName: \"kubernetes.io/projected/2b665100-4126-4836-af2c-a48494cfa6a0-kube-api-access-qrxpr\") pod \"maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.608357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.608316 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.610139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.610118 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.630518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.630484 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrxpr\" (UniqueName: \"kubernetes.io/projected/2b665100-4126-4836-af2c-a48494cfa6a0-kube-api-access-qrxpr\") pod \"maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.656871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.656845 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:43.978668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:43.978640 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk"] Oct 30 13:24:43.979728 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:24:43.979694 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b665100_4126_4836_af2c_a48494cfa6a0.slice/crio-02620f89f40acdc4a69830daa78a51eb742461058156c08907dc29ac60f6d70b WatchSource:0}: Error finding container 02620f89f40acdc4a69830daa78a51eb742461058156c08907dc29ac60f6d70b: Status 404 returned error can't find the container with id 02620f89f40acdc4a69830daa78a51eb742461058156c08907dc29ac60f6d70b Oct 30 13:24:44.123848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:44.121213 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:44.393408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:44.393365 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" event={"ID":"2b665100-4126-4836-af2c-a48494cfa6a0","Type":"ContainerStarted","Data":"02620f89f40acdc4a69830daa78a51eb742461058156c08907dc29ac60f6d70b"} Oct 30 13:24:45.403719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:45.403626 5049 generic.go:356] "Generic (PLEG): container finished" podID="2b665100-4126-4836-af2c-a48494cfa6a0" containerID="113399acf81ecb74a9f23a3ba314bde3e2f8bbd0136459329b764ad281b8cd89" exitCode=0 Oct 30 13:24:45.403719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:45.403666 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" event={"ID":"2b665100-4126-4836-af2c-a48494cfa6a0","Type":"ContainerDied","Data":"113399acf81ecb74a9f23a3ba314bde3e2f8bbd0136459329b764ad281b8cd89"} Oct 30 13:24:45.498975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:45.498939 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:46.063541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.061264 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:46.599086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.599051 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:46.760795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.760699 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b665100-4126-4836-af2c-a48494cfa6a0-kube-api-access-qrxpr\" (UniqueName: \"kubernetes.io/projected/2b665100-4126-4836-af2c-a48494cfa6a0-kube-api-access-qrxpr\") pod \"2b665100-4126-4836-af2c-a48494cfa6a0\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " Oct 30 13:24:46.760967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.760944 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-xks-ca\") pod \"2b665100-4126-4836-af2c-a48494cfa6a0\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " Oct 30 13:24:46.761087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.761067 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-openshift-ca\") pod \"2b665100-4126-4836-af2c-a48494cfa6a0\" (UID: \"2b665100-4126-4836-af2c-a48494cfa6a0\") " Oct 30 13:24:46.761461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.761417 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-openshift-ca" pod "2b665100-4126-4836-af2c-a48494cfa6a0" (UID: "2b665100-4126-4836-af2c-a48494cfa6a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:46.761790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.761753 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:46.763128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.763104 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-xks-ca" pod "2b665100-4126-4836-af2c-a48494cfa6a0" (UID: "2b665100-4126-4836-af2c-a48494cfa6a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:46.763128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.763117 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b665100-4126-4836-af2c-a48494cfa6a0-kube-api-access-qrxpr" pod "2b665100-4126-4836-af2c-a48494cfa6a0" (UID: "2b665100-4126-4836-af2c-a48494cfa6a0"). InnerVolumeSpecName "kube-api-access-qrxpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:46.863169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.863137 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b665100-4126-4836-af2c-a48494cfa6a0-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:46.863169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:46.863167 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrxpr\" (UniqueName: \"kubernetes.io/projected/2b665100-4126-4836-af2c-a48494cfa6a0-kube-api-access-qrxpr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:24:47.425512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:47.425479 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" Oct 30 13:24:47.425709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:47.425505 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk" event={"ID":"2b665100-4126-4836-af2c-a48494cfa6a0","Type":"ContainerDied","Data":"02620f89f40acdc4a69830daa78a51eb742461058156c08907dc29ac60f6d70b"} Oct 30 13:24:47.425709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:47.425591 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02620f89f40acdc4a69830daa78a51eb742461058156c08907dc29ac60f6d70b" Oct 30 13:24:48.059247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:48.059204 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:49.565629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:49.565590 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:49.825455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:49.825289 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:51.746793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:51.746758 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:51.752732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:51.752706 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:51.982747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:51.982716 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:52.552989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:52.552949 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:24:53.858418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:53.858377 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx"] Oct 30 13:24:53.891624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:53.891584 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx"] Oct 30 13:24:53.975149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:53.975039 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:54.205298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:54.205216 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:56.207650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:56.207613 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:56.679364 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:24:56.679302 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:24:58.074690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:58.074655 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:58.864268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:58.864210 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" podUID="d622eb69-bf74-4c55-89fe-18d963542243" containerName="payload-processing" containerID="cri-o://fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7" gracePeriod=25 Oct 30 13:24:58.892599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:58.892563 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" podUID="640a2237-2051-4ba3-8aac-5c617e742bb3" containerName="payload-pre-processing" containerID="cri-o://98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77" gracePeriod=25 Oct 30 13:24:59.849018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:24:59.848984 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:01.580707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:01.580662 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:03.539281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:03.539235 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:04.008262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:04.008218 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" podUID="d622eb69-bf74-4c55-89fe-18d963542243" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:25:04.008478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:04.008230 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" podUID="640a2237-2051-4ba3-8aac-5c617e742bb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:25:05.313339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:05.313289 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:07.013820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:07.013787 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:07.569279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:07.569244 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:07.881693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:07.881602 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:07.954529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:07.954494 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk"] Oct 30 13:25:07.959864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:07.959831 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77254d-r5xtk"] Oct 30 13:25:08.048269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:08.048233 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:08.456938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:08.456899 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:09.346669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:09.346633 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b665100-4126-4836-af2c-a48494cfa6a0" path="/var/lib/kubelet/pods/2b665100-4126-4836-af2c-a48494cfa6a0/volumes" Oct 30 13:25:09.932523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:09.932474 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:11.702439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:11.702405 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:13.423399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:13.422175 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:14.006841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:14.006796 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" podUID="d622eb69-bf74-4c55-89fe-18d963542243" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:25:14.007029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:14.006807 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" podUID="640a2237-2051-4ba3-8aac-5c617e742bb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:25:15.504142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.504105 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:15.871670 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.871640 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj"] Oct 30 13:25:15.872488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.872469 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2b665100-4126-4836-af2c-a48494cfa6a0" containerName="revoke-keys" Oct 30 13:25:15.872569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.872489 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b665100-4126-4836-af2c-a48494cfa6a0" containerName="revoke-keys" Oct 30 13:25:15.872656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.872646 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b665100-4126-4836-af2c-a48494cfa6a0" containerName="revoke-keys" Oct 30 13:25:15.876899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.876879 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:15.896448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.896425 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4651e8-openshift-default-dockercfg-v9dl8\"" Oct 30 13:25:15.911763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.911735 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj"] Oct 30 13:25:15.960349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.960284 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-podinfo\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:15.960542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.960434 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-envoy\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:15.960618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.960571 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30514d49-dc56-4c0a-bbfd-775dc42dc550-istiod-ca-cert\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:15.960667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.960638 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:15.960710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.960690 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-token\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:15.960875 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.960854 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-credential-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:15.961001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.960984 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-data\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:15.961083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.961023 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-certs\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:15.961144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.961080 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzh5c\" (UniqueName: \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-kube-api-access-tzh5c\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:15.995141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:15.995104 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:16.062539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.062505 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-data\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.062739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.062554 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-certs\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.062739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.062614 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzh5c\" (UniqueName: \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-kube-api-access-tzh5c\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.062739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.062645 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-podinfo\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.062739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.062711 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-envoy\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.062967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.062821 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30514d49-dc56-4c0a-bbfd-775dc42dc550-istiod-ca-cert\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.062967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.062861 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.062967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.062917 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-data\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.062967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.062921 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-token\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.063183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.062979 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-certs\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.063183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.063127 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-credential-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.063572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.063547 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-credential-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.063703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.063682 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.064001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.063978 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30514d49-dc56-4c0a-bbfd-775dc42dc550-istiod-ca-cert\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.065347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.065310 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-podinfo\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.065502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.065484 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-envoy\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.088545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.088523 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-token\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.088859 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.088833 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzh5c\" (UniqueName: \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-kube-api-access-tzh5c\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.186636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.186499 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:16.192047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.192025 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:16.314332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.309870 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm"] Oct 30 13:25:16.314332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.310062 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" containerID="cri-o://2c219d33a9c85ad4ae11999c8338464c86dd3fcef2010499600317f32166704b" gracePeriod=30 Oct 30 13:25:16.380289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.380249 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:16.576500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.567460 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj"] Oct 30 13:25:16.594696 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:25:16.594647 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30514d49_dc56_4c0a_bbfd_775dc42dc550.slice/crio-1b9ed3ce5c262f2cd7f9a8a8778323d2e6923691aae52ed70adcb817c8c0e619 WatchSource:0}: Error finding container 1b9ed3ce5c262f2cd7f9a8a8778323d2e6923691aae52ed70adcb817c8c0e619: Status 404 returned error can't find the container with id 1b9ed3ce5c262f2cd7f9a8a8778323d2e6923691aae52ed70adcb817c8c0e619 Oct 30 13:25:16.598427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.598391 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:25:16.598545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.598472 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:25:16.598545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.598505 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:25:16.739188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.736587 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" event={"ID":"30514d49-dc56-4c0a-bbfd-775dc42dc550","Type":"ContainerStarted","Data":"fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d"} Oct 30 13:25:16.739188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.736656 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" event={"ID":"30514d49-dc56-4c0a-bbfd-775dc42dc550","Type":"ContainerStarted","Data":"1b9ed3ce5c262f2cd7f9a8a8778323d2e6923691aae52ed70adcb817c8c0e619"} Oct 30 13:25:16.795571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.795510 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" podStartSLOduration=1.795495638 podStartE2EDuration="1.795495638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:16.791199725 +0000 UTC m=+2023.606394150" watchObservedRunningTime="2026-10-30 13:25:16.795495638 +0000 UTC m=+2023.610690065" Oct 30 13:25:16.883891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.883809 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8"] Oct 30 13:25:16.888698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.888676 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:16.901902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.901875 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:25:16.901902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.901883 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:25:16.917433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.916969 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8"] Oct 30 13:25:16.996368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.995043 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnl77\" (UniqueName: \"kubernetes.io/projected/dcab1f52-814b-4a85-abc5-2ba0f324cec9-kube-api-access-nnl77\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:16.996368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.995182 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:16.996368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:16.995271 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:17.103975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.099593 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl77\" (UniqueName: \"kubernetes.io/projected/dcab1f52-814b-4a85-abc5-2ba0f324cec9-kube-api-access-nnl77\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:17.103975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.099677 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:17.103975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.099725 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:17.104417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.104377 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:17.137235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.137160 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:17.149354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.149301 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnl77\" (UniqueName: \"kubernetes.io/projected/dcab1f52-814b-4a85-abc5-2ba0f324cec9-kube-api-access-nnl77\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:17.192951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.192914 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:17.194465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.194416 5049 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 13:25:17.194563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.194472 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 13:25:17.205954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.205928 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:17.556340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.556294 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8"] Oct 30 13:25:17.694569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.694539 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:17.779119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:17.776542 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" event={"ID":"dcab1f52-814b-4a85-abc5-2ba0f324cec9","Type":"ContainerStarted","Data":"f1b3689508ed83b9d7357e8919256c94ec793456cd62bfd6e214a2a2d68bf6e1"} Oct 30 13:25:18.193012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:18.192981 5049 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 13:25:18.193254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:18.193229 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 13:25:18.786520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:18.786482 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" event={"ID":"dcab1f52-814b-4a85-abc5-2ba0f324cec9","Type":"ContainerStarted","Data":"ad2facc231d6a7fd8736eadd5dba460444a7e7dd65ed8145abf22e1b1c4f9147"} Oct 30 13:25:18.839516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:18.839466 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" podStartSLOduration=2.839449054 podStartE2EDuration="2.839449054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:18.827870129 +0000 UTC m=+2025.643064556" watchObservedRunningTime="2026-10-30 13:25:18.839449054 +0000 UTC m=+2025.654643476" Oct 30 13:25:19.193467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.192869 5049 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 13:25:19.193467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.192946 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 13:25:19.295065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.295033 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:25:19.401277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.401247 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:25:19.467344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.467280 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d622eb69-bf74-4c55-89fe-18d963542243-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d622eb69-bf74-4c55-89fe-18d963542243-plugins-config-volume\") pod \"d622eb69-bf74-4c55-89fe-18d963542243\" (UID: \"d622eb69-bf74-4c55-89fe-18d963542243\") " Oct 30 13:25:19.467510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.467436 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d622eb69-bf74-4c55-89fe-18d963542243-kube-api-access-jkcpc\" (UniqueName: \"kubernetes.io/projected/d622eb69-bf74-4c55-89fe-18d963542243-kube-api-access-jkcpc\") pod \"d622eb69-bf74-4c55-89fe-18d963542243\" (UID: \"d622eb69-bf74-4c55-89fe-18d963542243\") " Oct 30 13:25:19.467901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.467880 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d622eb69-bf74-4c55-89fe-18d963542243-plugins-config-volume" pod "d622eb69-bf74-4c55-89fe-18d963542243" (UID: "d622eb69-bf74-4c55-89fe-18d963542243"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:19.468248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.468227 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d622eb69-bf74-4c55-89fe-18d963542243-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:19.470287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.470261 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d622eb69-bf74-4c55-89fe-18d963542243-kube-api-access-jkcpc" pod "d622eb69-bf74-4c55-89fe-18d963542243" (UID: "d622eb69-bf74-4c55-89fe-18d963542243"). InnerVolumeSpecName "kube-api-access-jkcpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:19.569413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.569297 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/640a2237-2051-4ba3-8aac-5c617e742bb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640a2237-2051-4ba3-8aac-5c617e742bb3-plugins-config-volume\") pod \"640a2237-2051-4ba3-8aac-5c617e742bb3\" (UID: \"640a2237-2051-4ba3-8aac-5c617e742bb3\") " Oct 30 13:25:19.569565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.569460 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640a2237-2051-4ba3-8aac-5c617e742bb3-kube-api-access-glt86\" (UniqueName: \"kubernetes.io/projected/640a2237-2051-4ba3-8aac-5c617e742bb3-kube-api-access-glt86\") pod \"640a2237-2051-4ba3-8aac-5c617e742bb3\" (UID: \"640a2237-2051-4ba3-8aac-5c617e742bb3\") " Oct 30 13:25:19.569795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.569733 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/640a2237-2051-4ba3-8aac-5c617e742bb3-plugins-config-volume" pod "640a2237-2051-4ba3-8aac-5c617e742bb3" (UID: "640a2237-2051-4ba3-8aac-5c617e742bb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:19.570250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.570224 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640a2237-2051-4ba3-8aac-5c617e742bb3-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:19.570390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.570254 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkcpc\" (UniqueName: \"kubernetes.io/projected/d622eb69-bf74-4c55-89fe-18d963542243-kube-api-access-jkcpc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:19.571871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.571843 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640a2237-2051-4ba3-8aac-5c617e742bb3-kube-api-access-glt86" pod "640a2237-2051-4ba3-8aac-5c617e742bb3" (UID: "640a2237-2051-4ba3-8aac-5c617e742bb3"). InnerVolumeSpecName "kube-api-access-glt86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:19.669578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.669540 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2"] Oct 30 13:25:19.670356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.670332 5049 cpu_manager.go:414] "containerMap: removing container" podUID="640a2237-2051-4ba3-8aac-5c617e742bb3" containerName="payload-pre-processing" Oct 30 13:25:19.670356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.670351 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640a2237-2051-4ba3-8aac-5c617e742bb3" containerName="payload-pre-processing" Oct 30 13:25:19.670450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.670391 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d622eb69-bf74-4c55-89fe-18d963542243" containerName="payload-processing" Oct 30 13:25:19.670450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.670396 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d622eb69-bf74-4c55-89fe-18d963542243" containerName="payload-processing" Oct 30 13:25:19.670535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.670525 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="640a2237-2051-4ba3-8aac-5c617e742bb3" containerName="payload-pre-processing" Oct 30 13:25:19.670567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.670544 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d622eb69-bf74-4c55-89fe-18d963542243" containerName="payload-processing" Oct 30 13:25:19.671972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.671951 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glt86\" (UniqueName: \"kubernetes.io/projected/640a2237-2051-4ba3-8aac-5c617e742bb3-kube-api-access-glt86\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:19.674573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.674552 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.690237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.690205 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-ff4285-openshift-default-dockercfg-4t4rj\"" Oct 30 13:25:19.741347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.705069 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:19.741347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.719971 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2"] Oct 30 13:25:19.789836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.789797 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.790280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.789928 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.790280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.790031 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.790280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.790120 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.790280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.790154 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.790280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.790178 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715c24ba-f9e7-4389-ba41-59b7ea89e582-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.790280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.790225 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.790663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.790312 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhcm\" (UniqueName: \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-kube-api-access-wwhcm\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.790663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.790356 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.849846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.849746 5049 generic.go:356] "Generic (PLEG): container finished" podID="d622eb69-bf74-4c55-89fe-18d963542243" containerID="fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7" exitCode=0 Oct 30 13:25:19.849846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.849793 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" Oct 30 13:25:19.850091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.849908 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" event={"ID":"d622eb69-bf74-4c55-89fe-18d963542243","Type":"ContainerDied","Data":"fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7"} Oct 30 13:25:19.850091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.849963 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx" event={"ID":"d622eb69-bf74-4c55-89fe-18d963542243","Type":"ContainerDied","Data":"161b74d53d5518cd8d824434b50f2dd3f133837124631ebc4dd73a2283087d96"} Oct 30 13:25:19.850091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.849991 5049 scope.go:122] "RemoveContainer" containerID="fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7" Oct 30 13:25:19.857440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.857413 5049 generic.go:356] "Generic (PLEG): container finished" podID="dcab1f52-814b-4a85-abc5-2ba0f324cec9" containerID="ad2facc231d6a7fd8736eadd5dba460444a7e7dd65ed8145abf22e1b1c4f9147" exitCode=0 Oct 30 13:25:19.857661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.857538 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" event={"ID":"dcab1f52-814b-4a85-abc5-2ba0f324cec9","Type":"ContainerDied","Data":"ad2facc231d6a7fd8736eadd5dba460444a7e7dd65ed8145abf22e1b1c4f9147"} Oct 30 13:25:19.862813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.862777 5049 generic.go:356] "Generic (PLEG): container finished" podID="640a2237-2051-4ba3-8aac-5c617e742bb3" containerID="98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77" exitCode=0 Oct 30 13:25:19.862925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.862889 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" event={"ID":"640a2237-2051-4ba3-8aac-5c617e742bb3","Type":"ContainerDied","Data":"98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77"} Oct 30 13:25:19.862925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.862921 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" event={"ID":"640a2237-2051-4ba3-8aac-5c617e742bb3","Type":"ContainerDied","Data":"df1d99df050ce7bc2eb4c16063b974da80632156b87645c6524ef27fbe7cd9f7"} Oct 30 13:25:19.863055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.862975 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx" Oct 30 13:25:19.874214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.873968 5049 scope.go:122] "RemoveContainer" containerID="fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7" Oct 30 13:25:19.874436 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:25:19.874409 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7\": container with ID starting with fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7 not found: ID does not exist" containerID="fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7" Oct 30 13:25:19.874519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.874449 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7"} err="failed to get container status \"fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7\": rpc error: code = NotFound desc = could not find container \"fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7\": container with ID starting with fa11fc45d94c1dc17a2f0b90703c6c1e038b8672ff93d6c6fc2474b502eeb8d7 not found: ID does not exist" Oct 30 13:25:19.874519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.874473 5049 scope.go:122] "RemoveContainer" containerID="98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77" Oct 30 13:25:19.886116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.886096 5049 scope.go:122] "RemoveContainer" containerID="98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77" Oct 30 13:25:19.886460 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:25:19.886432 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77\": container with ID starting with 98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77 not found: ID does not exist" containerID="98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77" Oct 30 13:25:19.886554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.886462 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77"} err="failed to get container status \"98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77\": rpc error: code = NotFound desc = could not find container \"98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77\": container with ID starting with 98ac647d698bd555b87e1a241d90522491bec37ea40e719000149f499db8db77 not found: ID does not exist" Oct 30 13:25:19.891678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.891649 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.891781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.891759 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.891856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.891792 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.891856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.891818 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715c24ba-f9e7-4389-ba41-59b7ea89e582-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.891963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.891867 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.892015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.891945 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhcm\" (UniqueName: \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-kube-api-access-wwhcm\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.892291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.892264 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.892394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.892376 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.892462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.892410 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.892522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.892511 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.892692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.892664 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.892912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.892892 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.892976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.892936 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.893291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.893276 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715c24ba-f9e7-4389-ba41-59b7ea89e582-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.894600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.894557 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.894824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.894803 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.938671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.938639 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhcm\" (UniqueName: \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-kube-api-access-wwhcm\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.938879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.938860 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:19.973342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.957835 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx"] Oct 30 13:25:19.973342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.968744 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77254d-84d4595f4b-hflpx"] Oct 30 13:25:19.989937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:19.989832 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:20.002722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.002693 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx"] Oct 30 13:25:20.033890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.020222 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77254d-7688484799-pzgnx"] Oct 30 13:25:20.192939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.192910 5049 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 13:25:20.193136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.192967 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 13:25:20.356074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.356038 5049 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:25:20.356224 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.356095 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:25:20.369380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.369341 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2"] Oct 30 13:25:20.370061 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:25:20.369828 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715c24ba_f9e7_4389_ba41_59b7ea89e582.slice/crio-9871ad7aaa0aa319e0f9579b648a9f6b4a3baed904f3ae1f304a81ca5473f4ef WatchSource:0}: Error finding container 9871ad7aaa0aa319e0f9579b648a9f6b4a3baed904f3ae1f304a81ca5473f4ef: Status 404 returned error can't find the container with id 9871ad7aaa0aa319e0f9579b648a9f6b4a3baed904f3ae1f304a81ca5473f4ef Oct 30 13:25:20.373335 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.373212 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:25:20.373454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.373424 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:25:20.373523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.373479 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:25:20.878079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.878043 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" event={"ID":"715c24ba-f9e7-4389-ba41-59b7ea89e582","Type":"ContainerStarted","Data":"380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5"} Oct 30 13:25:20.878702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.878098 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" event={"ID":"715c24ba-f9e7-4389-ba41-59b7ea89e582","Type":"ContainerStarted","Data":"9871ad7aaa0aa319e0f9579b648a9f6b4a3baed904f3ae1f304a81ca5473f4ef"} Oct 30 13:25:20.956409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.956338 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" podStartSLOduration=1.9563061880000001 podStartE2EDuration="1.956306188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:20.950421767 +0000 UTC m=+2027.765616190" watchObservedRunningTime="2026-10-30 13:25:20.956306188 +0000 UTC m=+2027.771500614" Oct 30 13:25:20.990135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:20.990104 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:21.200347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.200273 5049 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 13:25:21.200520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.200347 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 13:25:21.270214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.269843 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:21.399632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.399596 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="640a2237-2051-4ba3-8aac-5c617e742bb3" path="/var/lib/kubelet/pods/640a2237-2051-4ba3-8aac-5c617e742bb3/volumes" Oct 30 13:25:21.400573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.400548 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d622eb69-bf74-4c55-89fe-18d963542243" path="/var/lib/kubelet/pods/d622eb69-bf74-4c55-89fe-18d963542243/volumes" Oct 30 13:25:21.478357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.478313 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcab1f52-814b-4a85-abc5-2ba0f324cec9-kube-api-access-nnl77\" (UniqueName: \"kubernetes.io/projected/dcab1f52-814b-4a85-abc5-2ba0f324cec9-kube-api-access-nnl77\") pod \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " Oct 30 13:25:21.478489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.478380 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-xks-ca\") pod \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " Oct 30 13:25:21.478552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.478526 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-openshift-ca\") pod \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\" (UID: \"dcab1f52-814b-4a85-abc5-2ba0f324cec9\") " Oct 30 13:25:21.479621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.479592 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-openshift-ca" pod "dcab1f52-814b-4a85-abc5-2ba0f324cec9" (UID: "dcab1f52-814b-4a85-abc5-2ba0f324cec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:21.480416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.480394 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:21.494420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.485496 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-xks-ca" pod "dcab1f52-814b-4a85-abc5-2ba0f324cec9" (UID: "dcab1f52-814b-4a85-abc5-2ba0f324cec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:21.494420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.485783 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcab1f52-814b-4a85-abc5-2ba0f324cec9-kube-api-access-nnl77" pod "dcab1f52-814b-4a85-abc5-2ba0f324cec9" (UID: "dcab1f52-814b-4a85-abc5-2ba0f324cec9"). InnerVolumeSpecName "kube-api-access-nnl77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:21.594403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.584913 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:21.594403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.590816 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnl77\" (UniqueName: \"kubernetes.io/projected/dcab1f52-814b-4a85-abc5-2ba0f324cec9-kube-api-access-nnl77\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:21.594403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.590846 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcab1f52-814b-4a85-abc5-2ba0f324cec9-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:21.894069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.894033 5049 generic.go:356] "Generic (PLEG): container finished" podID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerID="2c219d33a9c85ad4ae11999c8338464c86dd3fcef2010499600317f32166704b" exitCode=0 Oct 30 13:25:21.895426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.894131 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" event={"ID":"6eacaf2d-bdf9-4f8c-918e-44f1910a7467","Type":"ContainerDied","Data":"2c219d33a9c85ad4ae11999c8338464c86dd3fcef2010499600317f32166704b"} Oct 30 13:25:21.895426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.894211 5049 scope.go:122] "RemoveContainer" containerID="93780e70c673bbe26c2f511141359f5e8064a8b310a8eb705ef9a38c34a6129f" Oct 30 13:25:21.901430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.901314 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" event={"ID":"dcab1f52-814b-4a85-abc5-2ba0f324cec9","Type":"ContainerDied","Data":"f1b3689508ed83b9d7357e8919256c94ec793456cd62bfd6e214a2a2d68bf6e1"} Oct 30 13:25:21.901501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.901461 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b3689508ed83b9d7357e8919256c94ec793456cd62bfd6e214a2a2d68bf6e1" Oct 30 13:25:21.901551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.901518 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8" Oct 30 13:25:21.994068 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:25:21.993903 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93780e70c673bbe26c2f511141359f5e8064a8b310a8eb705ef9a38c34a6129f\": container with ID starting with 93780e70c673bbe26c2f511141359f5e8064a8b310a8eb705ef9a38c34a6129f not found: ID does not exist" containerID="93780e70c673bbe26c2f511141359f5e8064a8b310a8eb705ef9a38c34a6129f" Oct 30 13:25:21.994225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.994091 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 13:25:21.994225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.994134 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" podUID="715c24ba-f9e7-4389-ba41-59b7ea89e582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 13:25:21.997812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:21.997785 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:25:22.115353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.105895 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-kube-api-access-jw4nh\" (UniqueName: \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-kube-api-access-jw4nh\") pod \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " Oct 30 13:25:22.115353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.106043 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-credential-socket\") pod \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " Oct 30 13:25:22.115353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.106094 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-socket\") pod \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " Oct 30 13:25:22.115353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.106311 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-token\" (UniqueName: \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-token\") pod \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " Oct 30 13:25:22.115353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.106375 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-podinfo\") pod \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " Oct 30 13:25:22.115353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.106405 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istiod-ca-cert\") pod \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " Oct 30 13:25:22.115353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.106443 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-envoy\") pod \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " Oct 30 13:25:22.115353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.106481 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-data\") pod \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " Oct 30 13:25:22.115353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.106559 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-certs\") pod \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\" (UID: \"6eacaf2d-bdf9-4f8c-918e-44f1910a7467\") " Oct 30 13:25:22.134516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.119625 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-socket" pod "6eacaf2d-bdf9-4f8c-918e-44f1910a7467" (UID: "6eacaf2d-bdf9-4f8c-918e-44f1910a7467"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:22.134516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.120123 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istiod-ca-cert" pod "6eacaf2d-bdf9-4f8c-918e-44f1910a7467" (UID: "6eacaf2d-bdf9-4f8c-918e-44f1910a7467"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:22.134516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.120454 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-credential-socket" pod "6eacaf2d-bdf9-4f8c-918e-44f1910a7467" (UID: "6eacaf2d-bdf9-4f8c-918e-44f1910a7467"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:22.134516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.134431 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-data" pod "6eacaf2d-bdf9-4f8c-918e-44f1910a7467" (UID: "6eacaf2d-bdf9-4f8c-918e-44f1910a7467"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:22.170614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.157886 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-certs" pod "6eacaf2d-bdf9-4f8c-918e-44f1910a7467" (UID: "6eacaf2d-bdf9-4f8c-918e-44f1910a7467"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:22.170614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.168269 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-token" pod "6eacaf2d-bdf9-4f8c-918e-44f1910a7467" (UID: "6eacaf2d-bdf9-4f8c-918e-44f1910a7467"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:22.170614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.169211 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-podinfo" pod "6eacaf2d-bdf9-4f8c-918e-44f1910a7467" (UID: "6eacaf2d-bdf9-4f8c-918e-44f1910a7467"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:22.170614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.170505 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-envoy" pod "6eacaf2d-bdf9-4f8c-918e-44f1910a7467" (UID: "6eacaf2d-bdf9-4f8c-918e-44f1910a7467"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:22.170614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.170552 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-kube-api-access-jw4nh" pod "6eacaf2d-bdf9-4f8c-918e-44f1910a7467" (UID: "6eacaf2d-bdf9-4f8c-918e-44f1910a7467"). InnerVolumeSpecName "kube-api-access-jw4nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:22.192866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.192829 5049 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 13:25:22.193055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.192907 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 13:25:22.249351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.249018 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw4nh\" (UniqueName: \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-kube-api-access-jw4nh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:22.256348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.249680 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:22.256348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.249711 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:22.256348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.249728 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:22.256348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.249743 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:22.256348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.249757 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:22.256348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.249771 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:22.256348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.249783 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:22.256348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.249795 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eacaf2d-bdf9-4f8c-918e-44f1910a7467-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:22.933344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.932933 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" Oct 30 13:25:22.933859 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.933470 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm" event={"ID":"6eacaf2d-bdf9-4f8c-918e-44f1910a7467","Type":"ContainerDied","Data":"5b3e43fc6180a7dbb4031dc492b20d1e15069d9752a642c0c842b6d63295eb60"} Oct 30 13:25:22.933859 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.933527 5049 scope.go:122] "RemoveContainer" containerID="2c219d33a9c85ad4ae11999c8338464c86dd3fcef2010499600317f32166704b" Oct 30 13:25:22.990881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.990846 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 13:25:22.991079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:22.990905 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" podUID="715c24ba-f9e7-4389-ba41-59b7ea89e582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 13:25:23.002679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:23.002650 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm"] Oct 30 13:25:23.024345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:23.014751 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-db9gm"] Oct 30 13:25:23.192677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:23.192598 5049 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 13:25:23.192677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:23.192647 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 13:25:23.353513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:23.353471 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" path="/var/lib/kubelet/pods/6eacaf2d-bdf9-4f8c-918e-44f1910a7467/volumes" Oct 30 13:25:23.599735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:23.599143 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:23.990714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:23.990621 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 13:25:23.990714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:23.990675 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" podUID="715c24ba-f9e7-4389-ba41-59b7ea89e582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 13:25:24.198907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:24.198874 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:24.199391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:24.199371 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:24.200077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:24.200059 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:25:24.991249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:24.991206 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 13:25:24.991857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:24.991270 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" podUID="715c24ba-f9e7-4389-ba41-59b7ea89e582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 13:25:25.458058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:25.443967 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:25.990481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:25.990446 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 13:25:25.990637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:25.990498 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" podUID="715c24ba-f9e7-4389-ba41-59b7ea89e582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 13:25:26.680971 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:25:26.680932 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:25:27.123650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:27.123619 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:27.124291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:27.124269 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:27.125365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:27.125303 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:25:27.369540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:27.369510 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:27.528873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:27.528786 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:27.685074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:27.685047 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:27.983816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:27.983767 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:29.354406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:29.354353 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:31.004345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.004299 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:31.204490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.204454 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:31.500383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.500347 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w"] Oct 30 13:25:31.501540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.501512 5049 cpu_manager.go:414] "containerMap: removing container" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" Oct 30 13:25:31.501540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.501539 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" Oct 30 13:25:31.501706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.501555 5049 cpu_manager.go:414] "containerMap: removing container" podUID="dcab1f52-814b-4a85-abc5-2ba0f324cec9" containerName="revoke-keys" Oct 30 13:25:31.501706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.501562 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcab1f52-814b-4a85-abc5-2ba0f324cec9" containerName="revoke-keys" Oct 30 13:25:31.501790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.501779 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" Oct 30 13:25:31.501850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.501805 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" Oct 30 13:25:31.501850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.501817 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcab1f52-814b-4a85-abc5-2ba0f324cec9" containerName="revoke-keys" Oct 30 13:25:31.506538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.506515 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:31.520065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.520037 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:25:31.528267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.528243 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w"] Oct 30 13:25:31.529817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.529774 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:25:31.661944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.661901 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167051bd-vc57w\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:31.662158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.661956 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167051bd-vc57w\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:31.662158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.662100 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlc5s\" (UniqueName: \"kubernetes.io/projected/52a1f076-8806-4777-bf33-bb4460d9735e-kube-api-access-qlc5s\") pod \"maas-api-revoke-sub-167051bd-vc57w\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:31.764103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.764017 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167051bd-vc57w\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:31.764103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.764060 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167051bd-vc57w\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:31.764385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.764132 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlc5s\" (UniqueName: \"kubernetes.io/projected/52a1f076-8806-4777-bf33-bb4460d9735e-kube-api-access-qlc5s\") pod \"maas-api-revoke-sub-167051bd-vc57w\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:31.764752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.764729 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167051bd-vc57w\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:31.766495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.766472 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167051bd-vc57w\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:31.786075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.786048 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlc5s\" (UniqueName: \"kubernetes.io/projected/52a1f076-8806-4777-bf33-bb4460d9735e-kube-api-access-qlc5s\") pod \"maas-api-revoke-sub-167051bd-vc57w\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:31.826020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:31.825981 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:32.147429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:32.147393 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w"] Oct 30 13:25:32.152175 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:25:32.152142 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a1f076_8806_4777_bf33_bb4460d9735e.slice/crio-a44b42a445e6ab21498a574a935e7e27ba78bac316b7c10bc834270b229476d0 WatchSource:0}: Error finding container a44b42a445e6ab21498a574a935e7e27ba78bac316b7c10bc834270b229476d0: Status 404 returned error can't find the container with id a44b42a445e6ab21498a574a935e7e27ba78bac316b7c10bc834270b229476d0 Oct 30 13:25:33.012905 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:33.012847 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:33.098775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:33.098733 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" event={"ID":"52a1f076-8806-4777-bf33-bb4460d9735e","Type":"ContainerStarted","Data":"afd5d1e8b389b6ecd2ac1117b3b8ccc6d47daca0612fa6e1e9d8ed56eef52648"} Oct 30 13:25:33.098966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:33.098836 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" event={"ID":"52a1f076-8806-4777-bf33-bb4460d9735e","Type":"ContainerStarted","Data":"a44b42a445e6ab21498a574a935e7e27ba78bac316b7c10bc834270b229476d0"} Oct 30 13:25:33.135093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:33.135032 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" podStartSLOduration=2.135014468 podStartE2EDuration="2.135014468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:33.134260889 +0000 UTC m=+2039.949455312" watchObservedRunningTime="2026-10-30 13:25:33.135014468 +0000 UTC m=+2039.950208894" Oct 30 13:25:34.108579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:34.108540 5049 generic.go:356] "Generic (PLEG): container finished" podID="52a1f076-8806-4777-bf33-bb4460d9735e" containerID="afd5d1e8b389b6ecd2ac1117b3b8ccc6d47daca0612fa6e1e9d8ed56eef52648" exitCode=0 Oct 30 13:25:34.109020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:34.108650 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" event={"ID":"52a1f076-8806-4777-bf33-bb4460d9735e","Type":"ContainerDied","Data":"afd5d1e8b389b6ecd2ac1117b3b8ccc6d47daca0612fa6e1e9d8ed56eef52648"} Oct 30 13:25:34.992309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:34.992272 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:35.326415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.326387 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:35.463489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.463456 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-xks-ca\") pod \"52a1f076-8806-4777-bf33-bb4460d9735e\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " Oct 30 13:25:35.463698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.463680 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a1f076-8806-4777-bf33-bb4460d9735e-kube-api-access-qlc5s\" (UniqueName: \"kubernetes.io/projected/52a1f076-8806-4777-bf33-bb4460d9735e-kube-api-access-qlc5s\") pod \"52a1f076-8806-4777-bf33-bb4460d9735e\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " Oct 30 13:25:35.463875 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.463856 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-openshift-ca\") pod \"52a1f076-8806-4777-bf33-bb4460d9735e\" (UID: \"52a1f076-8806-4777-bf33-bb4460d9735e\") " Oct 30 13:25:35.464260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.464206 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-openshift-ca" pod "52a1f076-8806-4777-bf33-bb4460d9735e" (UID: "52a1f076-8806-4777-bf33-bb4460d9735e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:35.464723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.464704 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:35.466074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.466047 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-xks-ca" pod "52a1f076-8806-4777-bf33-bb4460d9735e" (UID: "52a1f076-8806-4777-bf33-bb4460d9735e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:35.466190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.466164 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a1f076-8806-4777-bf33-bb4460d9735e-kube-api-access-qlc5s" pod "52a1f076-8806-4777-bf33-bb4460d9735e" (UID: "52a1f076-8806-4777-bf33-bb4460d9735e"). InnerVolumeSpecName "kube-api-access-qlc5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:35.566778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.566745 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlc5s\" (UniqueName: \"kubernetes.io/projected/52a1f076-8806-4777-bf33-bb4460d9735e-kube-api-access-qlc5s\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:35.566778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.566774 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a1f076-8806-4777-bf33-bb4460d9735e-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:35.633455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.633417 5049 scope.go:122] "RemoveContainer" containerID="a7da29c23a64f277f4b3955cfe8804a01ab2901ec0d120a2bd1e0731b16c58b1" Oct 30 13:25:35.649586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.649476 5049 scope.go:122] "RemoveContainer" containerID="a9ae8b318b266c57f5b97cc9efc0bb6427a5db91a887db245ebd272bd8d7df74" Oct 30 13:25:35.660956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:35.660928 5049 scope.go:122] "RemoveContainer" containerID="6453ec9cf78511ef9610220e079ffba00ccb0be6092ea83d0165b7e5d673c9f7" Oct 30 13:25:36.133644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:36.133612 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" Oct 30 13:25:36.133644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:36.133625 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w" event={"ID":"52a1f076-8806-4777-bf33-bb4460d9735e","Type":"ContainerDied","Data":"a44b42a445e6ab21498a574a935e7e27ba78bac316b7c10bc834270b229476d0"} Oct 30 13:25:36.133849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:36.133680 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a44b42a445e6ab21498a574a935e7e27ba78bac316b7c10bc834270b229476d0" Oct 30 13:25:36.820346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:36.818853 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:38.545543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:38.545140 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:40.518119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:40.518036 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:40.679184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:40.679152 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b"] Oct 30 13:25:40.679411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:40.679389 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" containerID="cri-o://198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6" gracePeriod=30 Oct 30 13:25:42.288506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:42.288472 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:43.984833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:43.984793 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:45.880199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:45.880061 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:46.016483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.016454 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:25:46.146519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.146475 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-credential-socket\") pod \"dafd548e-8a90-4a57-8046-c5731787e45a\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " Oct 30 13:25:46.146699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.146548 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-istio-token\" (UniqueName: \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-istio-token\") pod \"dafd548e-8a90-4a57-8046-c5731787e45a\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " Oct 30 13:25:46.146699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.146612 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-certs\") pod \"dafd548e-8a90-4a57-8046-c5731787e45a\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " Oct 30 13:25:46.146827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.146757 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dafd548e-8a90-4a57-8046-c5731787e45a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd548e-8a90-4a57-8046-c5731787e45a-istio-podinfo\") pod \"dafd548e-8a90-4a57-8046-c5731787e45a\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " Oct 30 13:25:46.146890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.146823 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafd548e-8a90-4a57-8046-c5731787e45a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd548e-8a90-4a57-8046-c5731787e45a-istiod-ca-cert\") pod \"dafd548e-8a90-4a57-8046-c5731787e45a\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " Oct 30 13:25:46.146969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.146942 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-data\") pod \"dafd548e-8a90-4a57-8046-c5731787e45a\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " Oct 30 13:25:46.146969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.146823 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-credential-socket" pod "dafd548e-8a90-4a57-8046-c5731787e45a" (UID: "dafd548e-8a90-4a57-8046-c5731787e45a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:46.147079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.147000 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-certs" pod "dafd548e-8a90-4a57-8046-c5731787e45a" (UID: "dafd548e-8a90-4a57-8046-c5731787e45a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:46.147079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.147009 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-kube-api-access-wx6mz\" (UniqueName: \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-kube-api-access-wx6mz\") pod \"dafd548e-8a90-4a57-8046-c5731787e45a\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " Oct 30 13:25:46.147186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.147136 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-envoy\") pod \"dafd548e-8a90-4a57-8046-c5731787e45a\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " Oct 30 13:25:46.147241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.147190 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-socket\") pod \"dafd548e-8a90-4a57-8046-c5731787e45a\" (UID: \"dafd548e-8a90-4a57-8046-c5731787e45a\") " Oct 30 13:25:46.147241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.147214 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-data" pod "dafd548e-8a90-4a57-8046-c5731787e45a" (UID: "dafd548e-8a90-4a57-8046-c5731787e45a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:46.147374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.147253 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafd548e-8a90-4a57-8046-c5731787e45a-istiod-ca-cert" pod "dafd548e-8a90-4a57-8046-c5731787e45a" (UID: "dafd548e-8a90-4a57-8046-c5731787e45a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:46.147579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.147528 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-socket" pod "dafd548e-8a90-4a57-8046-c5731787e45a" (UID: "dafd548e-8a90-4a57-8046-c5731787e45a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:46.148072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.148046 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.148181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.148078 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.148181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.148093 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.148181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.148109 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.148181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.148126 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd548e-8a90-4a57-8046-c5731787e45a-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.149465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.149426 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-istio-token" pod "dafd548e-8a90-4a57-8046-c5731787e45a" (UID: "dafd548e-8a90-4a57-8046-c5731787e45a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:46.149586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.149512 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dafd548e-8a90-4a57-8046-c5731787e45a-istio-podinfo" pod "dafd548e-8a90-4a57-8046-c5731787e45a" (UID: "dafd548e-8a90-4a57-8046-c5731787e45a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:46.149586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.149537 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-kube-api-access-wx6mz" pod "dafd548e-8a90-4a57-8046-c5731787e45a" (UID: "dafd548e-8a90-4a57-8046-c5731787e45a"). InnerVolumeSpecName "kube-api-access-wx6mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:46.150166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.150142 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-envoy" pod "dafd548e-8a90-4a57-8046-c5731787e45a" (UID: "dafd548e-8a90-4a57-8046-c5731787e45a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:46.236215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.226031 5049 generic.go:356] "Generic (PLEG): container finished" podID="dafd548e-8a90-4a57-8046-c5731787e45a" containerID="198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6" exitCode=0 Oct 30 13:25:46.236215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.226241 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" event={"ID":"dafd548e-8a90-4a57-8046-c5731787e45a","Type":"ContainerDied","Data":"198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6"} Oct 30 13:25:46.236215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.226342 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" event={"ID":"dafd548e-8a90-4a57-8046-c5731787e45a","Type":"ContainerDied","Data":"c7fbc05d900df352ac19ced67dd660772f726065e0a6bc43a428fdaf618b0136"} Oct 30 13:25:46.236215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.226375 5049 scope.go:122] "RemoveContainer" containerID="198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6" Oct 30 13:25:46.236215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.226549 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b" Oct 30 13:25:46.245024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.245004 5049 scope.go:122] "RemoveContainer" containerID="198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6" Oct 30 13:25:46.245370 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:25:46.245343 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6\": container with ID starting with 198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6 not found: ID does not exist" containerID="198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6" Oct 30 13:25:46.245451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.245381 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6"} err="failed to get container status \"198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6\": rpc error: code = NotFound desc = could not find container \"198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6\": container with ID starting with 198870cdfa76d5dc3ce562054140afa59caacf57de6c5262aeb082aeed07a4e6 not found: ID does not exist" Oct 30 13:25:46.249716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.249688 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.249813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.249720 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd548e-8a90-4a57-8046-c5731787e45a-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.249813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.249735 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx6mz\" (UniqueName: \"kubernetes.io/projected/dafd548e-8a90-4a57-8046-c5731787e45a-kube-api-access-wx6mz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.249813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.249748 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd548e-8a90-4a57-8046-c5731787e45a-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:46.279814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.279772 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b"] Oct 30 13:25:46.283927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:46.283901 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-fgd6b"] Oct 30 13:25:47.339465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:47.339432 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" path="/var/lib/kubelet/pods/dafd548e-8a90-4a57-8046-c5731787e45a/volumes" Oct 30 13:25:47.676988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:47.675900 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:49.435709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:49.430011 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:51.436288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:51.436239 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:53.307808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.307767 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:53.779424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.779339 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj"] Oct 30 13:25:53.780461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.780443 5049 cpu_manager.go:414] "containerMap: removing container" podUID="52a1f076-8806-4777-bf33-bb4460d9735e" containerName="revoke-keys" Oct 30 13:25:53.780547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.780464 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a1f076-8806-4777-bf33-bb4460d9735e" containerName="revoke-keys" Oct 30 13:25:53.780547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.780481 5049 cpu_manager.go:414] "containerMap: removing container" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" Oct 30 13:25:53.780547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.780486 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eacaf2d-bdf9-4f8c-918e-44f1910a7467" containerName="istio-proxy" Oct 30 13:25:53.780547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.780498 5049 cpu_manager.go:414] "containerMap: removing container" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" Oct 30 13:25:53.780547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.780503 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" Oct 30 13:25:53.780722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.780658 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a1f076-8806-4777-bf33-bb4460d9735e" containerName="revoke-keys" Oct 30 13:25:53.780722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.780668 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafd548e-8a90-4a57-8046-c5731787e45a" containerName="istio-proxy" Oct 30 13:25:53.786073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.786047 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:53.803639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.803612 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj"] Oct 30 13:25:53.808888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.808868 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:25:53.809030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.809010 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:25:53.887932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.887896 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w986c\" (UniqueName: \"kubernetes.io/projected/d7f7af8e-8869-4195-a8be-10de819b2e5c-kube-api-access-w986c\") pod \"maas-api-revoke-sub-c90a6f86-hj4rj\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:53.888098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.887941 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c90a6f86-hj4rj\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:53.888098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.888032 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c90a6f86-hj4rj\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:53.989582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.989549 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w986c\" (UniqueName: \"kubernetes.io/projected/d7f7af8e-8869-4195-a8be-10de819b2e5c-kube-api-access-w986c\") pod \"maas-api-revoke-sub-c90a6f86-hj4rj\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:53.989582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.989588 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c90a6f86-hj4rj\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:53.989815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.989771 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c90a6f86-hj4rj\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:53.990415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.990397 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c90a6f86-hj4rj\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:53.991865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:53.991847 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c90a6f86-hj4rj\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:54.011145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:54.011116 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w986c\" (UniqueName: \"kubernetes.io/projected/d7f7af8e-8869-4195-a8be-10de819b2e5c-kube-api-access-w986c\") pod \"maas-api-revoke-sub-c90a6f86-hj4rj\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:54.101196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:54.101169 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:54.415998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:54.415970 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj"] Oct 30 13:25:54.417855 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:25:54.417821 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f7af8e_8869_4195_a8be_10de819b2e5c.slice/crio-dd1ae7e2a056d80c6106e784a72f0732943c0effa8e051bd725b0ce5bf744d12 WatchSource:0}: Error finding container dd1ae7e2a056d80c6106e784a72f0732943c0effa8e051bd725b0ce5bf744d12: Status 404 returned error can't find the container with id dd1ae7e2a056d80c6106e784a72f0732943c0effa8e051bd725b0ce5bf744d12 Oct 30 13:25:55.148303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:55.148208 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:55.339331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:55.339282 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" event={"ID":"d7f7af8e-8869-4195-a8be-10de819b2e5c","Type":"ContainerStarted","Data":"daa9ae4250e45dd9c80a5b873f18391ef8848ab4f155744c3fc19cded5a6cb99"} Oct 30 13:25:55.339502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:55.339358 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" event={"ID":"d7f7af8e-8869-4195-a8be-10de819b2e5c","Type":"ContainerStarted","Data":"dd1ae7e2a056d80c6106e784a72f0732943c0effa8e051bd725b0ce5bf744d12"} Oct 30 13:25:55.368100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:55.368049 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" podStartSLOduration=2.36803552 podStartE2EDuration="2.36803552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:55.365712864 +0000 UTC m=+2062.180907289" watchObservedRunningTime="2026-10-30 13:25:55.36803552 +0000 UTC m=+2062.183229943" Oct 30 13:25:56.337983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:56.337950 5049 generic.go:356] "Generic (PLEG): container finished" podID="d7f7af8e-8869-4195-a8be-10de819b2e5c" containerID="daa9ae4250e45dd9c80a5b873f18391ef8848ab4f155744c3fc19cded5a6cb99" exitCode=0 Oct 30 13:25:56.338462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:56.337996 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" event={"ID":"d7f7af8e-8869-4195-a8be-10de819b2e5c","Type":"ContainerDied","Data":"daa9ae4250e45dd9c80a5b873f18391ef8848ab4f155744c3fc19cded5a6cb99"} Oct 30 13:25:56.679307 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:25:56.679220 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:25:57.051080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.051014 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:57.525972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.525943 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:57.654740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.654645 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-openshift-ca\") pod \"d7f7af8e-8869-4195-a8be-10de819b2e5c\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " Oct 30 13:25:57.654905 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.654848 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f7af8e-8869-4195-a8be-10de819b2e5c-kube-api-access-w986c\" (UniqueName: \"kubernetes.io/projected/d7f7af8e-8869-4195-a8be-10de819b2e5c-kube-api-access-w986c\") pod \"d7f7af8e-8869-4195-a8be-10de819b2e5c\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " Oct 30 13:25:57.655007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.654987 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-xks-ca\") pod \"d7f7af8e-8869-4195-a8be-10de819b2e5c\" (UID: \"d7f7af8e-8869-4195-a8be-10de819b2e5c\") " Oct 30 13:25:57.655119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.655093 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-openshift-ca" pod "d7f7af8e-8869-4195-a8be-10de819b2e5c" (UID: "d7f7af8e-8869-4195-a8be-10de819b2e5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:57.655705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.655676 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:57.657195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.657166 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f7af8e-8869-4195-a8be-10de819b2e5c-kube-api-access-w986c" pod "d7f7af8e-8869-4195-a8be-10de819b2e5c" (UID: "d7f7af8e-8869-4195-a8be-10de819b2e5c"). InnerVolumeSpecName "kube-api-access-w986c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:57.657492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.657470 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-xks-ca" pod "d7f7af8e-8869-4195-a8be-10de819b2e5c" (UID: "d7f7af8e-8869-4195-a8be-10de819b2e5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:57.757613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.757579 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f7af8e-8869-4195-a8be-10de819b2e5c-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:57.757613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:57.757607 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w986c\" (UniqueName: \"kubernetes.io/projected/d7f7af8e-8869-4195-a8be-10de819b2e5c-kube-api-access-w986c\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:25:58.355920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:58.355887 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" Oct 30 13:25:58.355920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:58.355911 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj" event={"ID":"d7f7af8e-8869-4195-a8be-10de819b2e5c","Type":"ContainerDied","Data":"dd1ae7e2a056d80c6106e784a72f0732943c0effa8e051bd725b0ce5bf744d12"} Oct 30 13:25:58.356137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:58.355964 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd1ae7e2a056d80c6106e784a72f0732943c0effa8e051bd725b0ce5bf744d12" Oct 30 13:25:58.886377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:25:58.885115 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:00.640459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:00.640426 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:02.569400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:02.569357 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:04.115396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.115364 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r"] Oct 30 13:26:04.116240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.116186 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d7f7af8e-8869-4195-a8be-10de819b2e5c" containerName="revoke-keys" Oct 30 13:26:04.116240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.116201 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f7af8e-8869-4195-a8be-10de819b2e5c" containerName="revoke-keys" Oct 30 13:26:04.116384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.116359 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f7af8e-8869-4195-a8be-10de819b2e5c" containerName="revoke-keys" Oct 30 13:26:04.120442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.120425 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.150402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.150375 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r"] Oct 30 13:26:04.154560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.154537 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:26:04.154704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.154640 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:26:04.254126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.254084 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqml\" (UniqueName: \"kubernetes.io/projected/943b7f36-93c3-4815-8904-1e545470612d-kube-api-access-xbqml\") pod \"maas-api-revoke-sub-99802cce-r8d8r\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.254298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.254265 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/943b7f36-93c3-4815-8904-1e545470612d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99802cce-r8d8r\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.254411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.254394 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/943b7f36-93c3-4815-8904-1e545470612d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99802cce-r8d8r\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.355892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.355859 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqml\" (UniqueName: \"kubernetes.io/projected/943b7f36-93c3-4815-8904-1e545470612d-kube-api-access-xbqml\") pod \"maas-api-revoke-sub-99802cce-r8d8r\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.356036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.356004 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/943b7f36-93c3-4815-8904-1e545470612d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99802cce-r8d8r\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.356196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.356106 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/943b7f36-93c3-4815-8904-1e545470612d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99802cce-r8d8r\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.356785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.356762 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/943b7f36-93c3-4815-8904-1e545470612d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99802cce-r8d8r\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.358405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.358387 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/943b7f36-93c3-4815-8904-1e545470612d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99802cce-r8d8r\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.380060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.380000 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqml\" (UniqueName: \"kubernetes.io/projected/943b7f36-93c3-4815-8904-1e545470612d-kube-api-access-xbqml\") pod \"maas-api-revoke-sub-99802cce-r8d8r\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.413170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.413137 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:04.434703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.434659 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:04.764936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:04.764909 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r"] Oct 30 13:26:04.767550 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:26:04.767516 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943b7f36_93c3_4815_8904_1e545470612d.slice/crio-df313332dd19395fa4ac1b88eefa54998e79790848054cf6f0272c1c91df87e5 WatchSource:0}: Error finding container df313332dd19395fa4ac1b88eefa54998e79790848054cf6f0272c1c91df87e5: Status 404 returned error can't find the container with id df313332dd19395fa4ac1b88eefa54998e79790848054cf6f0272c1c91df87e5 Oct 30 13:26:05.301556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:05.301523 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:05.431127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:05.431089 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" event={"ID":"943b7f36-93c3-4815-8904-1e545470612d","Type":"ContainerStarted","Data":"d80d9494773d9cffac53fa507d536dfdada8dae9bfaa4ad70a8e564b1d6b2440"} Oct 30 13:26:05.431273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:05.431142 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" event={"ID":"943b7f36-93c3-4815-8904-1e545470612d","Type":"ContainerStarted","Data":"df313332dd19395fa4ac1b88eefa54998e79790848054cf6f0272c1c91df87e5"} Oct 30 13:26:05.450202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:05.450169 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:05.471583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:05.471525 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" podStartSLOduration=1.471508272 podStartE2EDuration="1.471508272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:05.466814593 +0000 UTC m=+2072.282009021" watchObservedRunningTime="2026-10-30 13:26:05.471508272 +0000 UTC m=+2072.286702711" Oct 30 13:26:05.810313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:05.810271 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:06.376152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:06.376096 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:06.470402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:06.470362 5049 generic.go:356] "Generic (PLEG): container finished" podID="943b7f36-93c3-4815-8904-1e545470612d" containerID="d80d9494773d9cffac53fa507d536dfdada8dae9bfaa4ad70a8e564b1d6b2440" exitCode=0 Oct 30 13:26:06.470891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:06.470860 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" event={"ID":"943b7f36-93c3-4815-8904-1e545470612d","Type":"ContainerDied","Data":"d80d9494773d9cffac53fa507d536dfdada8dae9bfaa4ad70a8e564b1d6b2440"} Oct 30 13:26:06.630374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:06.630226 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:07.688589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:07.688560 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:07.723641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:07.723611 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943b7f36-93c3-4815-8904-1e545470612d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/943b7f36-93c3-4815-8904-1e545470612d-maas-api-openshift-ca\") pod \"943b7f36-93c3-4815-8904-1e545470612d\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " Oct 30 13:26:07.723822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:07.723708 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943b7f36-93c3-4815-8904-1e545470612d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/943b7f36-93c3-4815-8904-1e545470612d-maas-api-xks-ca\") pod \"943b7f36-93c3-4815-8904-1e545470612d\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " Oct 30 13:26:07.723822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:07.723765 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943b7f36-93c3-4815-8904-1e545470612d-kube-api-access-xbqml\" (UniqueName: \"kubernetes.io/projected/943b7f36-93c3-4815-8904-1e545470612d-kube-api-access-xbqml\") pod \"943b7f36-93c3-4815-8904-1e545470612d\" (UID: \"943b7f36-93c3-4815-8904-1e545470612d\") " Oct 30 13:26:07.724094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:07.724066 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943b7f36-93c3-4815-8904-1e545470612d-maas-api-openshift-ca" pod "943b7f36-93c3-4815-8904-1e545470612d" (UID: "943b7f36-93c3-4815-8904-1e545470612d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:07.724660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:07.724628 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/943b7f36-93c3-4815-8904-1e545470612d-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:07.726362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:07.726307 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943b7f36-93c3-4815-8904-1e545470612d-maas-api-xks-ca" pod "943b7f36-93c3-4815-8904-1e545470612d" (UID: "943b7f36-93c3-4815-8904-1e545470612d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:07.726492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:07.726456 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943b7f36-93c3-4815-8904-1e545470612d-kube-api-access-xbqml" pod "943b7f36-93c3-4815-8904-1e545470612d" (UID: "943b7f36-93c3-4815-8904-1e545470612d"). InnerVolumeSpecName "kube-api-access-xbqml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:07.825984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:07.825952 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/943b7f36-93c3-4815-8904-1e545470612d-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:07.825984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:07.825982 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbqml\" (UniqueName: \"kubernetes.io/projected/943b7f36-93c3-4815-8904-1e545470612d-kube-api-access-xbqml\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:08.489138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:08.489108 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" event={"ID":"943b7f36-93c3-4815-8904-1e545470612d","Type":"ContainerDied","Data":"df313332dd19395fa4ac1b88eefa54998e79790848054cf6f0272c1c91df87e5"} Oct 30 13:26:08.489291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:08.489151 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r" Oct 30 13:26:08.489291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:08.489156 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df313332dd19395fa4ac1b88eefa54998e79790848054cf6f0272c1c91df87e5" Oct 30 13:26:08.635383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:08.628091 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:10.431364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:10.431293 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:12.457308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:12.457149 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:13.951806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:13.951765 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq"] Oct 30 13:26:13.952299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:13.951997 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" podUID="24f54e01-a626-4c94-9b8a-bb28250ceae2" containerName="maas-api" containerID="cri-o://52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18" gracePeriod=30 Oct 30 13:26:14.094525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.094475 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:14.300785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.300753 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:26:14.461396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.461286 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f54e01-a626-4c94-9b8a-bb28250ceae2-kube-api-access-w2rzn\" (UniqueName: \"kubernetes.io/projected/24f54e01-a626-4c94-9b8a-bb28250ceae2-kube-api-access-w2rzn\") pod \"24f54e01-a626-4c94-9b8a-bb28250ceae2\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " Oct 30 13:26:14.461396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.461369 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-metrics-certs\") pod \"24f54e01-a626-4c94-9b8a-bb28250ceae2\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " Oct 30 13:26:14.461681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.461413 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f54e01-a626-4c94-9b8a-bb28250ceae2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f54e01-a626-4c94-9b8a-bb28250ceae2-openshift-service-ca\") pod \"24f54e01-a626-4c94-9b8a-bb28250ceae2\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " Oct 30 13:26:14.461681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.461544 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-maas-api-tls\") pod \"24f54e01-a626-4c94-9b8a-bb28250ceae2\" (UID: \"24f54e01-a626-4c94-9b8a-bb28250ceae2\") " Oct 30 13:26:14.467669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.464034 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f54e01-a626-4c94-9b8a-bb28250ceae2-openshift-service-ca" pod "24f54e01-a626-4c94-9b8a-bb28250ceae2" (UID: "24f54e01-a626-4c94-9b8a-bb28250ceae2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:14.467669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.467263 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-maas-api-tls" pod "24f54e01-a626-4c94-9b8a-bb28250ceae2" (UID: "24f54e01-a626-4c94-9b8a-bb28250ceae2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:14.467669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.467410 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f54e01-a626-4c94-9b8a-bb28250ceae2-kube-api-access-w2rzn" pod "24f54e01-a626-4c94-9b8a-bb28250ceae2" (UID: "24f54e01-a626-4c94-9b8a-bb28250ceae2"). InnerVolumeSpecName "kube-api-access-w2rzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:14.490100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.483507 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-metrics-certs" pod "24f54e01-a626-4c94-9b8a-bb28250ceae2" (UID: "24f54e01-a626-4c94-9b8a-bb28250ceae2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:14.564343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.564136 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rzn\" (UniqueName: \"kubernetes.io/projected/24f54e01-a626-4c94-9b8a-bb28250ceae2-kube-api-access-w2rzn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:14.564343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.564166 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:14.564343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.564177 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f54e01-a626-4c94-9b8a-bb28250ceae2-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:14.564343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.564186 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f54e01-a626-4c94-9b8a-bb28250ceae2-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:14.594346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.593871 5049 generic.go:356] "Generic (PLEG): container finished" podID="24f54e01-a626-4c94-9b8a-bb28250ceae2" containerID="52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18" exitCode=0 Oct 30 13:26:14.594346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.594149 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" event={"ID":"24f54e01-a626-4c94-9b8a-bb28250ceae2","Type":"ContainerDied","Data":"52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18"} Oct 30 13:26:14.594346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.594200 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" event={"ID":"24f54e01-a626-4c94-9b8a-bb28250ceae2","Type":"ContainerDied","Data":"db59c76799c038718cc9c4bb12fd26660adf2bfed1f40bdeacf1edb8b5256b1f"} Oct 30 13:26:14.594346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.594229 5049 scope.go:122] "RemoveContainer" containerID="52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18" Oct 30 13:26:14.594721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.594412 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq" Oct 30 13:26:14.622253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.621939 5049 scope.go:122] "RemoveContainer" containerID="52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18" Oct 30 13:26:14.623050 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:26:14.622953 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18\": container with ID starting with 52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18 not found: ID does not exist" containerID="52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18" Oct 30 13:26:14.623050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.622992 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18"} err="failed to get container status \"52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18\": rpc error: code = NotFound desc = could not find container \"52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18\": container with ID starting with 52778c99b318a423093533b872a0aaa0679524a5301b90572c1bb7c3c602cb18 not found: ID does not exist" Oct 30 13:26:14.688364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.673090 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq"] Oct 30 13:26:14.725463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.691800 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:14.725463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.714806 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb271-676674db6-w4mhq"] Oct 30 13:26:14.796639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.796609 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8"] Oct 30 13:26:14.802868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.802843 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb271-f5gd8"] Oct 30 13:26:14.847097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:14.847065 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:15.316693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:15.316657 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:15.338989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:15.338959 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f54e01-a626-4c94-9b8a-bb28250ceae2" path="/var/lib/kubelet/pods/24f54e01-a626-4c94-9b8a-bb28250ceae2/volumes" Oct 30 13:26:15.339465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:15.339449 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcab1f52-814b-4a85-abc5-2ba0f324cec9" path="/var/lib/kubelet/pods/dcab1f52-814b-4a85-abc5-2ba0f324cec9/volumes" Oct 30 13:26:16.749225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.749196 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:16.861373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.861336 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l"] Oct 30 13:26:16.862159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.862144 5049 cpu_manager.go:414] "containerMap: removing container" podUID="943b7f36-93c3-4815-8904-1e545470612d" containerName="revoke-keys" Oct 30 13:26:16.862204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.862161 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943b7f36-93c3-4815-8904-1e545470612d" containerName="revoke-keys" Oct 30 13:26:16.862204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.862181 5049 cpu_manager.go:414] "containerMap: removing container" podUID="24f54e01-a626-4c94-9b8a-bb28250ceae2" containerName="maas-api" Oct 30 13:26:16.862204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.862187 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f54e01-a626-4c94-9b8a-bb28250ceae2" containerName="maas-api" Oct 30 13:26:16.862334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.862305 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="943b7f36-93c3-4815-8904-1e545470612d" containerName="revoke-keys" Oct 30 13:26:16.862334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.862316 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f54e01-a626-4c94-9b8a-bb28250ceae2" containerName="maas-api" Oct 30 13:26:16.867445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.867428 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:16.893687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.893660 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l"] Oct 30 13:26:16.910105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.910075 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:26:16.910105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.910087 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:26:16.970258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:16.970211 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:17.008602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.008461 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b6092f3-ffr4l\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:17.008602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.008565 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzq7z\" (UniqueName: \"kubernetes.io/projected/4e392228-99d1-4bc8-9567-a293c9e7a20a-kube-api-access-fzq7z\") pod \"maas-api-revoke-sub-0b6092f3-ffr4l\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:17.008849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.008775 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b6092f3-ffr4l\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:17.110285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.110215 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b6092f3-ffr4l\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:17.110502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.110410 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzq7z\" (UniqueName: \"kubernetes.io/projected/4e392228-99d1-4bc8-9567-a293c9e7a20a-kube-api-access-fzq7z\") pod \"maas-api-revoke-sub-0b6092f3-ffr4l\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:17.110616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.110598 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b6092f3-ffr4l\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:17.110998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.110962 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b6092f3-ffr4l\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:17.113118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.113097 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b6092f3-ffr4l\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:17.133369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.133298 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzq7z\" (UniqueName: \"kubernetes.io/projected/4e392228-99d1-4bc8-9567-a293c9e7a20a-kube-api-access-fzq7z\") pod \"maas-api-revoke-sub-0b6092f3-ffr4l\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:17.193945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.193905 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:17.682879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:17.682853 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l"] Oct 30 13:26:17.684793 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:26:17.684768 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e392228_99d1_4bc8_9567_a293c9e7a20a.slice/crio-7f86aee531f0dce09dcabb474e66d047f2c9796c2939863ba325b1196d2d3dcf WatchSource:0}: Error finding container 7f86aee531f0dce09dcabb474e66d047f2c9796c2939863ba325b1196d2d3dcf: Status 404 returned error can't find the container with id 7f86aee531f0dce09dcabb474e66d047f2c9796c2939863ba325b1196d2d3dcf Oct 30 13:26:18.528224 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:18.528130 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:18.649461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:18.648210 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" event={"ID":"4e392228-99d1-4bc8-9567-a293c9e7a20a","Type":"ContainerStarted","Data":"fcf84099b0c55dad51818b0fc7838729761b5b1da69ccb6383edf856f3ab951b"} Oct 30 13:26:18.649461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:18.648336 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" event={"ID":"4e392228-99d1-4bc8-9567-a293c9e7a20a","Type":"ContainerStarted","Data":"7f86aee531f0dce09dcabb474e66d047f2c9796c2939863ba325b1196d2d3dcf"} Oct 30 13:26:19.100042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:19.100006 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:19.260251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:19.260138 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:19.413684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:19.413651 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:19.659084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:19.659028 5049 generic.go:356] "Generic (PLEG): container finished" podID="4e392228-99d1-4bc8-9567-a293c9e7a20a" containerID="fcf84099b0c55dad51818b0fc7838729761b5b1da69ccb6383edf856f3ab951b" exitCode=0 Oct 30 13:26:19.659532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:19.659118 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" event={"ID":"4e392228-99d1-4bc8-9567-a293c9e7a20a","Type":"ContainerDied","Data":"fcf84099b0c55dad51818b0fc7838729761b5b1da69ccb6383edf856f3ab951b"} Oct 30 13:26:19.755280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:19.755244 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:20.106642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.106603 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7"] Oct 30 13:26:20.106870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.106802 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" podUID="a6f8396f-7488-447c-97f9-2073e0d6199c" containerName="discovery" containerID="cri-o://70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459" gracePeriod=30 Oct 30 13:26:20.429898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.429871 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:26:20.552507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.552474 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-token\" (UniqueName: \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-token\") pod \"a6f8396f-7488-447c-97f9-2073e0d6199c\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " Oct 30 13:26:20.552683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.552601 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-kubeconfig\") pod \"a6f8396f-7488-447c-97f9-2073e0d6199c\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " Oct 30 13:26:20.552683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.552662 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-ca-configmap\") pod \"a6f8396f-7488-447c-97f9-2073e0d6199c\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " Oct 30 13:26:20.552785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.552708 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-kube-api-access-6bp9r\" (UniqueName: \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-kube-api-access-6bp9r\") pod \"a6f8396f-7488-447c-97f9-2073e0d6199c\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " Oct 30 13:26:20.552856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.552817 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f8396f-7488-447c-97f9-2073e0d6199c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f8396f-7488-447c-97f9-2073e0d6199c-local-certs\") pod \"a6f8396f-7488-447c-97f9-2073e0d6199c\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " Oct 30 13:26:20.552955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.552937 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-cacerts\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-cacerts\") pod \"a6f8396f-7488-447c-97f9-2073e0d6199c\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " Oct 30 13:26:20.553030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.553018 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-dns-cert\") pod \"a6f8396f-7488-447c-97f9-2073e0d6199c\" (UID: \"a6f8396f-7488-447c-97f9-2073e0d6199c\") " Oct 30 13:26:20.553169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.553089 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-ca-configmap" pod "a6f8396f-7488-447c-97f9-2073e0d6199c" (UID: "a6f8396f-7488-447c-97f9-2073e0d6199c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:20.553856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.553816 5049 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-ca-configmap\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:20.555573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.555535 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-kubeconfig" pod "a6f8396f-7488-447c-97f9-2073e0d6199c" (UID: "a6f8396f-7488-447c-97f9-2073e0d6199c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:20.555697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.555602 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-kube-api-access-6bp9r" pod "a6f8396f-7488-447c-97f9-2073e0d6199c" (UID: "a6f8396f-7488-447c-97f9-2073e0d6199c"). InnerVolumeSpecName "kube-api-access-6bp9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:20.555697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.555618 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-cacerts" pod "a6f8396f-7488-447c-97f9-2073e0d6199c" (UID: "a6f8396f-7488-447c-97f9-2073e0d6199c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:20.555697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.555636 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-token" pod "a6f8396f-7488-447c-97f9-2073e0d6199c" (UID: "a6f8396f-7488-447c-97f9-2073e0d6199c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:20.556682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.556649 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-dns-cert" pod "a6f8396f-7488-447c-97f9-2073e0d6199c" (UID: "a6f8396f-7488-447c-97f9-2073e0d6199c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:20.556987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.556967 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f8396f-7488-447c-97f9-2073e0d6199c-local-certs" pod "a6f8396f-7488-447c-97f9-2073e0d6199c" (UID: "a6f8396f-7488-447c-97f9-2073e0d6199c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:20.655199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.655096 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:20.655199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.655132 5049 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-kubeconfig\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:20.655199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.655142 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bp9r\" (UniqueName: \"kubernetes.io/projected/a6f8396f-7488-447c-97f9-2073e0d6199c-kube-api-access-6bp9r\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:20.655199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.655153 5049 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f8396f-7488-447c-97f9-2073e0d6199c-local-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:20.655199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.655172 5049 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-cacerts\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:20.655199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.655180 5049 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a6f8396f-7488-447c-97f9-2073e0d6199c-istio-csr-dns-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:20.671067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.671028 5049 generic.go:356] "Generic (PLEG): container finished" podID="a6f8396f-7488-447c-97f9-2073e0d6199c" containerID="70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459" exitCode=0 Oct 30 13:26:20.671067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.671069 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" Oct 30 13:26:20.671726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.671110 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" event={"ID":"a6f8396f-7488-447c-97f9-2073e0d6199c","Type":"ContainerDied","Data":"70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459"} Oct 30 13:26:20.671726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.671174 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7" event={"ID":"a6f8396f-7488-447c-97f9-2073e0d6199c","Type":"ContainerDied","Data":"e9aa6803999dc0ecb9cadd13a5e78e75b3869338f3ce63185b72223afe06e1db"} Oct 30 13:26:20.671726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.671218 5049 scope.go:122] "RemoveContainer" containerID="70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459" Oct 30 13:26:20.692177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.692147 5049 scope.go:122] "RemoveContainer" containerID="70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459" Oct 30 13:26:20.692554 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:26:20.692534 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459\": container with ID starting with 70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459 not found: ID does not exist" containerID="70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459" Oct 30 13:26:20.692634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.692564 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459"} err="failed to get container status \"70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459\": rpc error: code = NotFound desc = could not find container \"70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459\": container with ID starting with 70bd9c31c9a283dab5c40a5fb462e0f0657a93a4c94c8bfbbca5759b0bf29459 not found: ID does not exist" Oct 30 13:26:20.724257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.724212 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7"] Oct 30 13:26:20.728605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.728575 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82sf7"] Oct 30 13:26:20.849085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.849056 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:20.963435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.963355 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-xks-ca\") pod \"4e392228-99d1-4bc8-9567-a293c9e7a20a\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " Oct 30 13:26:20.963605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.963470 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-openshift-ca\") pod \"4e392228-99d1-4bc8-9567-a293c9e7a20a\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " Oct 30 13:26:20.963752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.963737 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e392228-99d1-4bc8-9567-a293c9e7a20a-kube-api-access-fzq7z\" (UniqueName: \"kubernetes.io/projected/4e392228-99d1-4bc8-9567-a293c9e7a20a-kube-api-access-fzq7z\") pod \"4e392228-99d1-4bc8-9567-a293c9e7a20a\" (UID: \"4e392228-99d1-4bc8-9567-a293c9e7a20a\") " Oct 30 13:26:20.963836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.963803 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-openshift-ca" pod "4e392228-99d1-4bc8-9567-a293c9e7a20a" (UID: "4e392228-99d1-4bc8-9567-a293c9e7a20a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:20.964224 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.964206 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:20.965574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.965552 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-xks-ca" pod "4e392228-99d1-4bc8-9567-a293c9e7a20a" (UID: "4e392228-99d1-4bc8-9567-a293c9e7a20a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:20.965769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:20.965751 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e392228-99d1-4bc8-9567-a293c9e7a20a-kube-api-access-fzq7z" pod "4e392228-99d1-4bc8-9567-a293c9e7a20a" (UID: "4e392228-99d1-4bc8-9567-a293c9e7a20a"). InnerVolumeSpecName "kube-api-access-fzq7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:21.065570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:21.065532 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e392228-99d1-4bc8-9567-a293c9e7a20a-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:21.065570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:21.065564 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzq7z\" (UniqueName: \"kubernetes.io/projected/4e392228-99d1-4bc8-9567-a293c9e7a20a-kube-api-access-fzq7z\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:21.299472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:21.299442 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:21.339006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:21.338973 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f8396f-7488-447c-97f9-2073e0d6199c" path="/var/lib/kubelet/pods/a6f8396f-7488-447c-97f9-2073e0d6199c/volumes" Oct 30 13:26:21.683642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:21.683532 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" Oct 30 13:26:21.683642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:21.683593 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l" event={"ID":"4e392228-99d1-4bc8-9567-a293c9e7a20a","Type":"ContainerDied","Data":"7f86aee531f0dce09dcabb474e66d047f2c9796c2939863ba325b1196d2d3dcf"} Oct 30 13:26:21.684177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:21.683651 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f86aee531f0dce09dcabb474e66d047f2c9796c2939863ba325b1196d2d3dcf" Oct 30 13:26:21.995643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:21.995564 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:22.149541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:22.149505 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:22.266317 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:22.266236 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh"] Oct 30 13:26:22.266545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:22.266517 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerName="istio-proxy" containerID="cri-o://dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40" gracePeriod=30 Oct 30 13:26:22.332673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:22.332642 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:22.785019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:22.784935 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:23.153187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.153155 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9"] Oct 30 13:26:23.153932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.153913 5049 cpu_manager.go:414] "containerMap: removing container" podUID="a6f8396f-7488-447c-97f9-2073e0d6199c" containerName="discovery" Oct 30 13:26:23.153932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.153930 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f8396f-7488-447c-97f9-2073e0d6199c" containerName="discovery" Oct 30 13:26:23.154128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.153943 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4e392228-99d1-4bc8-9567-a293c9e7a20a" containerName="revoke-keys" Oct 30 13:26:23.154128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.153949 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e392228-99d1-4bc8-9567-a293c9e7a20a" containerName="revoke-keys" Oct 30 13:26:23.154128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.154096 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f8396f-7488-447c-97f9-2073e0d6199c" containerName="discovery" Oct 30 13:26:23.154128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.154104 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e392228-99d1-4bc8-9567-a293c9e7a20a" containerName="revoke-keys" Oct 30 13:26:23.159357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.159339 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.185062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.182812 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:26:23.185062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.183074 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:26:23.185970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.185937 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9"] Oct 30 13:26:23.311714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.311666 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.311948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.311836 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.311948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.311888 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8n44\" (UniqueName: \"kubernetes.io/projected/3ac9ac74-3545-48e1-ae41-89814ee3259b-kube-api-access-h8n44\") pod \"maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.415295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.415190 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8n44\" (UniqueName: \"kubernetes.io/projected/3ac9ac74-3545-48e1-ae41-89814ee3259b-kube-api-access-h8n44\") pod \"maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.415521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.415485 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.415668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.415647 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.416893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.416869 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.428374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.419464 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.438518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.438477 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8n44\" (UniqueName: \"kubernetes.io/projected/3ac9ac74-3545-48e1-ae41-89814ee3259b-kube-api-access-h8n44\") pod \"maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.475291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.475260 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:23.500412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.500368 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:23.805852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:23.805824 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9"] Oct 30 13:26:23.807169 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:26:23.807139 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac9ac74_3545_48e1_ae41_89814ee3259b.slice/crio-2d4f20f4e3e304acd3af8d69e19ed58c4af095f496bc7dcf827db2359c6d54a3 WatchSource:0}: Error finding container 2d4f20f4e3e304acd3af8d69e19ed58c4af095f496bc7dcf827db2359c6d54a3: Status 404 returned error can't find the container with id 2d4f20f4e3e304acd3af8d69e19ed58c4af095f496bc7dcf827db2359c6d54a3 Oct 30 13:26:24.717360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:24.717306 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" event={"ID":"3ac9ac74-3545-48e1-ae41-89814ee3259b","Type":"ContainerStarted","Data":"8718c51224b2c5afdd5ae3c6c61be77cec3a7a9b2ea3926cc7f897cda504e7f6"} Oct 30 13:26:24.717538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:24.717380 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" event={"ID":"3ac9ac74-3545-48e1-ae41-89814ee3259b","Type":"ContainerStarted","Data":"2d4f20f4e3e304acd3af8d69e19ed58c4af095f496bc7dcf827db2359c6d54a3"} Oct 30 13:26:24.757952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:24.757873 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" podStartSLOduration=1.7578589930000001 podStartE2EDuration="1.757858993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:24.753843324 +0000 UTC m=+2091.569037748" watchObservedRunningTime="2026-10-30 13:26:24.757858993 +0000 UTC m=+2091.573053418" Oct 30 13:26:24.898492 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:26:24.898462 5049 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac9ac74_3545_48e1_ae41_89814ee3259b.slice/crio-conmon-8718c51224b2c5afdd5ae3c6c61be77cec3a7a9b2ea3926cc7f897cda504e7f6.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:26:25.365588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:25.365544 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:25.731731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:25.731566 5049 generic.go:356] "Generic (PLEG): container finished" podID="3ac9ac74-3545-48e1-ae41-89814ee3259b" containerID="8718c51224b2c5afdd5ae3c6c61be77cec3a7a9b2ea3926cc7f897cda504e7f6" exitCode=0 Oct 30 13:26:25.731918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:25.731891 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" event={"ID":"3ac9ac74-3545-48e1-ae41-89814ee3259b","Type":"ContainerDied","Data":"8718c51224b2c5afdd5ae3c6c61be77cec3a7a9b2ea3926cc7f897cda504e7f6"} Oct 30 13:26:26.708548 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:26:26.708500 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:26:27.012643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.012611 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:27.025604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.025571 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac9ac74-3545-48e1-ae41-89814ee3259b-kube-api-access-h8n44\" (UniqueName: \"kubernetes.io/projected/3ac9ac74-3545-48e1-ae41-89814ee3259b-kube-api-access-h8n44\") pod \"3ac9ac74-3545-48e1-ae41-89814ee3259b\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " Oct 30 13:26:27.025800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.025674 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-xks-ca\") pod \"3ac9ac74-3545-48e1-ae41-89814ee3259b\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " Oct 30 13:26:27.025890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.025845 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-openshift-ca\") pod \"3ac9ac74-3545-48e1-ae41-89814ee3259b\" (UID: \"3ac9ac74-3545-48e1-ae41-89814ee3259b\") " Oct 30 13:26:27.026779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.026747 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-openshift-ca" pod "3ac9ac74-3545-48e1-ae41-89814ee3259b" (UID: "3ac9ac74-3545-48e1-ae41-89814ee3259b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:27.028564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.028540 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-xks-ca" pod "3ac9ac74-3545-48e1-ae41-89814ee3259b" (UID: "3ac9ac74-3545-48e1-ae41-89814ee3259b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:27.082478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.081577 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac9ac74-3545-48e1-ae41-89814ee3259b-kube-api-access-h8n44" pod "3ac9ac74-3545-48e1-ae41-89814ee3259b" (UID: "3ac9ac74-3545-48e1-ae41-89814ee3259b"). InnerVolumeSpecName "kube-api-access-h8n44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:27.131286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.129658 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.131286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.129692 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8n44\" (UniqueName: \"kubernetes.io/projected/3ac9ac74-3545-48e1-ae41-89814ee3259b-kube-api-access-h8n44\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.131286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.129722 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac9ac74-3545-48e1-ae41-89814ee3259b-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.135270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.135239 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:27.324708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.318793 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:27.467392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.465103 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp"] Oct 30 13:26:27.467392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.465548 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" containerID="cri-o://3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e" gracePeriod=30 Oct 30 13:26:27.543919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.543885 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:27.655628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.655469 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.751716 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-credential-socket\") pod \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.751768 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-certs\") pod \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.751831 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-envoy\") pod \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.751861 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-podinfo\") pod \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.751985 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-token\" (UniqueName: \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-token\") pod \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.752072 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-data\") pod \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.752115 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-socket\") pod \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.752157 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-kube-api-access-p7f8k\" (UniqueName: \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-kube-api-access-p7f8k\") pod \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.752268 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istiod-ca-cert\") pod \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\" (UID: \"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a\") " Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.753313 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istiod-ca-cert" pod "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" (UID: "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.753954 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-credential-socket" pod "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" (UID: "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.754175 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-data" pod "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" (UID: "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.754379 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-socket" pod "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" (UID: "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.755600 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-certs" pod "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" (UID: "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:27.760699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.758997 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-envoy" pod "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" (UID: "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:27.790680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.790642 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-token" pod "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" (UID: "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:27.790878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.790754 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-kube-api-access-p7f8k" pod "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" (UID: "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a"). InnerVolumeSpecName "kube-api-access-p7f8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:27.790878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.790825 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-podinfo" pod "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" (UID: "cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:26:27.831332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.831234 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" event={"ID":"3ac9ac74-3545-48e1-ae41-89814ee3259b","Type":"ContainerDied","Data":"2d4f20f4e3e304acd3af8d69e19ed58c4af095f496bc7dcf827db2359c6d54a3"} Oct 30 13:26:27.831332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.831291 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d4f20f4e3e304acd3af8d69e19ed58c4af095f496bc7dcf827db2359c6d54a3" Oct 30 13:26:27.831595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.831367 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9" Oct 30 13:26:27.834445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.834387 5049 generic.go:356] "Generic (PLEG): container finished" podID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerID="dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40" exitCode=0 Oct 30 13:26:27.834840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.834796 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" event={"ID":"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a","Type":"ContainerDied","Data":"dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40"} Oct 30 13:26:27.836421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.835012 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" event={"ID":"cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a","Type":"ContainerDied","Data":"affa42afeac4b89529309a2a06da6e7839ceee7148ba247bd8d45b83e7134899"} Oct 30 13:26:27.836421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.835055 5049 scope.go:122] "RemoveContainer" containerID="dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40" Oct 30 13:26:27.836421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.835228 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh" Oct 30 13:26:27.853585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.853492 5049 scope.go:122] "RemoveContainer" containerID="dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40" Oct 30 13:26:27.854909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.854885 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.855036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.854914 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.855036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.854927 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.855036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.854950 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.855036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.854962 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.855036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.854975 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7f8k\" (UniqueName: \"kubernetes.io/projected/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-kube-api-access-p7f8k\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.855036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.854989 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.855036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.855001 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.855036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.855015 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:27.855704 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:26:27.855598 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40\": container with ID starting with dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40 not found: ID does not exist" containerID="dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40" Oct 30 13:26:27.855704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.855638 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40"} err="failed to get container status \"dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40\": rpc error: code = NotFound desc = could not find container \"dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40\": container with ID starting with dbfb28efae9d047e8fbee80ad88d1a387c73fa8f9b15d0b71ae4c9ad2d138b40 not found: ID does not exist" Oct 30 13:26:27.902679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.902650 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh"] Oct 30 13:26:27.908397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.908370 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pmhmh"] Oct 30 13:26:27.965202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.965166 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8"] Oct 30 13:26:27.965933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.965915 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3ac9ac74-3545-48e1-ae41-89814ee3259b" containerName="revoke-keys" Oct 30 13:26:27.965933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.965932 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac9ac74-3545-48e1-ae41-89814ee3259b" containerName="revoke-keys" Oct 30 13:26:27.966093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.965953 5049 cpu_manager.go:414] "containerMap: removing container" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerName="istio-proxy" Oct 30 13:26:27.966093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.965959 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerName="istio-proxy" Oct 30 13:26:27.966093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.966093 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac9ac74-3545-48e1-ae41-89814ee3259b" containerName="revoke-keys" Oct 30 13:26:27.966268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.966106 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" containerName="istio-proxy" Oct 30 13:26:27.971097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.971072 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:27.984737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.984717 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:26:27.985015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.984995 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:26:27.999116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:27.999087 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8"] Oct 30 13:26:28.057655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.057619 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp66t\" (UniqueName: \"kubernetes.io/projected/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-kube-api-access-jp66t\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:28.057870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.057755 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:28.057943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.057927 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:28.088049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.087972 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:28.159473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.159443 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:28.159650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.159605 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp66t\" (UniqueName: \"kubernetes.io/projected/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-kube-api-access-jp66t\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:28.159748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.159729 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:28.160358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.160296 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:28.161967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.161943 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:28.180882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.180852 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp66t\" (UniqueName: \"kubernetes.io/projected/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-kube-api-access-jp66t\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:28.237969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.237940 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:28.298628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.298599 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:28.618951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.618919 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8"] Oct 30 13:26:28.620154 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:26:28.620128 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dd8cec_22ea_4f4d_8b08_f17869fe76d5.slice/crio-eb8cbdc54ec4876a1d49389df12b6a1f6974a1ecd3862b9a90e9abbde8159e41 WatchSource:0}: Error finding container eb8cbdc54ec4876a1d49389df12b6a1f6974a1ecd3862b9a90e9abbde8159e41: Status 404 returned error can't find the container with id eb8cbdc54ec4876a1d49389df12b6a1f6974a1ecd3862b9a90e9abbde8159e41 Oct 30 13:26:28.844382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:28.844316 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" event={"ID":"22dd8cec-22ea-4f4d-8b08-f17869fe76d5","Type":"ContainerStarted","Data":"eb8cbdc54ec4876a1d49389df12b6a1f6974a1ecd3862b9a90e9abbde8159e41"} Oct 30 13:26:29.340242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:29.340211 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a" path="/var/lib/kubelet/pods/cdb3d2a9-aadb-48a9-a2d7-4b5b97c7bf0a/volumes" Oct 30 13:26:29.382899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:29.382863 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:29.856944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:29.856908 5049 generic.go:356] "Generic (PLEG): container finished" podID="22dd8cec-22ea-4f4d-8b08-f17869fe76d5" containerID="b8c22acf5be1117e4ceef176421f8f260543df0e24f21c4938b48b5c07f2855e" exitCode=0 Oct 30 13:26:29.857359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:29.857000 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" event={"ID":"22dd8cec-22ea-4f4d-8b08-f17869fe76d5","Type":"ContainerDied","Data":"b8c22acf5be1117e4ceef176421f8f260543df0e24f21c4938b48b5c07f2855e"} Oct 30 13:26:31.043987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.043958 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:31.106330 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.106298 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-openshift-ca\") pod \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " Oct 30 13:26:31.106503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.106395 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-xks-ca\") pod \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " Oct 30 13:26:31.106700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.106677 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-kube-api-access-jp66t\" (UniqueName: \"kubernetes.io/projected/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-kube-api-access-jp66t\") pod \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\" (UID: \"22dd8cec-22ea-4f4d-8b08-f17869fe76d5\") " Oct 30 13:26:31.106700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.106687 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-openshift-ca" pod "22dd8cec-22ea-4f4d-8b08-f17869fe76d5" (UID: "22dd8cec-22ea-4f4d-8b08-f17869fe76d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:31.107447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.107413 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:31.110735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.110704 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-xks-ca" pod "22dd8cec-22ea-4f4d-8b08-f17869fe76d5" (UID: "22dd8cec-22ea-4f4d-8b08-f17869fe76d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:31.110844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.110782 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-kube-api-access-jp66t" pod "22dd8cec-22ea-4f4d-8b08-f17869fe76d5" (UID: "22dd8cec-22ea-4f4d-8b08-f17869fe76d5"). InnerVolumeSpecName "kube-api-access-jp66t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:31.209053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.209014 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp66t\" (UniqueName: \"kubernetes.io/projected/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-kube-api-access-jp66t\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:31.209053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.209053 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22dd8cec-22ea-4f4d-8b08-f17869fe76d5-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:31.326064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.326016 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:31.875783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.875748 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" Oct 30 13:26:31.875783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.875779 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8" event={"ID":"22dd8cec-22ea-4f4d-8b08-f17869fe76d5","Type":"ContainerDied","Data":"eb8cbdc54ec4876a1d49389df12b6a1f6974a1ecd3862b9a90e9abbde8159e41"} Oct 30 13:26:31.876049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:31.875838 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb8cbdc54ec4876a1d49389df12b6a1f6974a1ecd3862b9a90e9abbde8159e41" Oct 30 13:26:32.790201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.790171 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:26:32.834691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.834660 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-envoy\") pod \"eb0be57a-8d1b-4498-928c-48616b906a53\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " Oct 30 13:26:32.834895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.834878 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-data\") pod \"eb0be57a-8d1b-4498-928c-48616b906a53\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " Oct 30 13:26:32.834956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.834925 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-credential-socket\") pod \"eb0be57a-8d1b-4498-928c-48616b906a53\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " Oct 30 13:26:32.835001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.834955 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0be57a-8d1b-4498-928c-48616b906a53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0be57a-8d1b-4498-928c-48616b906a53-istiod-ca-cert\") pod \"eb0be57a-8d1b-4498-928c-48616b906a53\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " Oct 30 13:26:32.835001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.834992 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-istio-token\" (UniqueName: \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-istio-token\") pod \"eb0be57a-8d1b-4498-928c-48616b906a53\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " Oct 30 13:26:32.835101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.835066 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-kube-api-access-v2zdb\" (UniqueName: \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-kube-api-access-v2zdb\") pod \"eb0be57a-8d1b-4498-928c-48616b906a53\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " Oct 30 13:26:32.835622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.835216 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-certs\") pod \"eb0be57a-8d1b-4498-928c-48616b906a53\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " Oct 30 13:26:32.835622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.835250 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb0be57a-8d1b-4498-928c-48616b906a53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0be57a-8d1b-4498-928c-48616b906a53-istio-podinfo\") pod \"eb0be57a-8d1b-4498-928c-48616b906a53\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " Oct 30 13:26:32.835622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.835335 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-socket\") pod \"eb0be57a-8d1b-4498-928c-48616b906a53\" (UID: \"eb0be57a-8d1b-4498-928c-48616b906a53\") " Oct 30 13:26:32.835622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.835394 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-credential-socket" pod "eb0be57a-8d1b-4498-928c-48616b906a53" (UID: "eb0be57a-8d1b-4498-928c-48616b906a53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:32.835622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.835476 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0be57a-8d1b-4498-928c-48616b906a53-istiod-ca-cert" pod "eb0be57a-8d1b-4498-928c-48616b906a53" (UID: "eb0be57a-8d1b-4498-928c-48616b906a53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:32.835622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.835614 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-certs" pod "eb0be57a-8d1b-4498-928c-48616b906a53" (UID: "eb0be57a-8d1b-4498-928c-48616b906a53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:32.836087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.835783 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-socket" pod "eb0be57a-8d1b-4498-928c-48616b906a53" (UID: "eb0be57a-8d1b-4498-928c-48616b906a53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:32.836087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.835835 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-data" pod "eb0be57a-8d1b-4498-928c-48616b906a53" (UID: "eb0be57a-8d1b-4498-928c-48616b906a53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:32.836205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.836121 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:32.836205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.836142 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:32.836205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.836156 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0be57a-8d1b-4498-928c-48616b906a53-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:32.836205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.836171 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:32.836205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.836185 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:32.837421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.837387 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-envoy" pod "eb0be57a-8d1b-4498-928c-48616b906a53" (UID: "eb0be57a-8d1b-4498-928c-48616b906a53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:32.837962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.837936 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb0be57a-8d1b-4498-928c-48616b906a53-istio-podinfo" pod "eb0be57a-8d1b-4498-928c-48616b906a53" (UID: "eb0be57a-8d1b-4498-928c-48616b906a53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:26:32.838079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.837993 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-istio-token" pod "eb0be57a-8d1b-4498-928c-48616b906a53" (UID: "eb0be57a-8d1b-4498-928c-48616b906a53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:32.838307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.838289 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-kube-api-access-v2zdb" pod "eb0be57a-8d1b-4498-928c-48616b906a53" (UID: "eb0be57a-8d1b-4498-928c-48616b906a53"). InnerVolumeSpecName "kube-api-access-v2zdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:32.886156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.886122 5049 generic.go:356] "Generic (PLEG): container finished" podID="eb0be57a-8d1b-4498-928c-48616b906a53" containerID="3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e" exitCode=0 Oct 30 13:26:32.886418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.886398 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" event={"ID":"eb0be57a-8d1b-4498-928c-48616b906a53","Type":"ContainerDied","Data":"3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e"} Oct 30 13:26:32.886512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.886456 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" event={"ID":"eb0be57a-8d1b-4498-928c-48616b906a53","Type":"ContainerDied","Data":"d3b1887537528c7f92b2369ab7e956a72fb931db0efe62a8f300fe3fd18d4e11"} Oct 30 13:26:32.886512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.886489 5049 scope.go:122] "RemoveContainer" containerID="3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e" Oct 30 13:26:32.886659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.886644 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp" Oct 30 13:26:32.901080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.901056 5049 scope.go:122] "RemoveContainer" containerID="f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd" Oct 30 13:26:32.913435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.913382 5049 scope.go:122] "RemoveContainer" containerID="3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e" Oct 30 13:26:32.913743 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:26:32.913714 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e\": container with ID starting with 3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e not found: ID does not exist" containerID="3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e" Oct 30 13:26:32.913819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.913750 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e"} err="failed to get container status \"3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e\": rpc error: code = NotFound desc = could not find container \"3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e\": container with ID starting with 3a5aa6447e16d98b7fd04ab8952001bed61985f38053d63b6d6f182befa1ff8e not found: ID does not exist" Oct 30 13:26:32.913819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.913769 5049 scope.go:122] "RemoveContainer" containerID="f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd" Oct 30 13:26:32.914034 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:26:32.914013 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd\": container with ID starting with f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd not found: ID does not exist" containerID="f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd" Oct 30 13:26:32.914102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.914040 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd"} err="failed to get container status \"f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd\": rpc error: code = NotFound desc = could not find container \"f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd\": container with ID starting with f6ffa85e324de0e76c838c0bf0c92c9ffffc6bffdd80202e3a310ad717f08bfd not found: ID does not exist" Oct 30 13:26:32.935736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.935711 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp"] Oct 30 13:26:32.937783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.937762 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:32.937783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.937783 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2zdb\" (UniqueName: \"kubernetes.io/projected/eb0be57a-8d1b-4498-928c-48616b906a53-kube-api-access-v2zdb\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:32.937952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.937793 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0be57a-8d1b-4498-928c-48616b906a53-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:32.937952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.937803 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0be57a-8d1b-4498-928c-48616b906a53-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:32.944432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:32.944406 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-31721c-openshift-default-7685767649-hbppp"] Oct 30 13:26:33.110264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:33.110206 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:33.340615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:33.340584 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" path="/var/lib/kubelet/pods/eb0be57a-8d1b-4498-928c-48616b906a53/volumes" Oct 30 13:26:33.532548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:33.532427 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:33.744283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:33.744243 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:26:33.748299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:33.748272 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:26:34.116742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.116710 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58"] Oct 30 13:26:34.117509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.117491 5049 cpu_manager.go:414] "containerMap: removing container" podUID="22dd8cec-22ea-4f4d-8b08-f17869fe76d5" containerName="revoke-keys" Oct 30 13:26:34.117509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.117510 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dd8cec-22ea-4f4d-8b08-f17869fe76d5" containerName="revoke-keys" Oct 30 13:26:34.117662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.117541 5049 cpu_manager.go:414] "containerMap: removing container" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" Oct 30 13:26:34.117662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.117546 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" Oct 30 13:26:34.117779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.117671 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" Oct 30 13:26:34.117779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.117683 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" Oct 30 13:26:34.117779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.117694 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dd8cec-22ea-4f4d-8b08-f17869fe76d5" containerName="revoke-keys" Oct 30 13:26:34.123174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.123152 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.137766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.137743 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:26:34.152506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.152471 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:26:34.155673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.154828 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58"] Oct 30 13:26:34.245359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.245310 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p"] Oct 30 13:26:34.257023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.256984 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e57fb3-94t58\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.257167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.257073 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e57fb3-94t58\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.257230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.257157 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qgj\" (UniqueName: \"kubernetes.io/projected/a2978447-8e53-461f-9a9e-527f2dfc1452-kube-api-access-47qgj\") pod \"maas-api-revoke-sub-91e57fb3-94t58\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.359127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.359092 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e57fb3-94t58\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.359359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.359156 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e57fb3-94t58\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.359359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.359211 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qgj\" (UniqueName: \"kubernetes.io/projected/a2978447-8e53-461f-9a9e-527f2dfc1452-kube-api-access-47qgj\") pod \"maas-api-revoke-sub-91e57fb3-94t58\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.360519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.360490 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e57fb3-94t58\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.379900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.379828 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e57fb3-94t58\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.383304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.383272 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qgj\" (UniqueName: \"kubernetes.io/projected/a2978447-8e53-461f-9a9e-527f2dfc1452-kube-api-access-47qgj\") pod \"maas-api-revoke-sub-91e57fb3-94t58\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.437829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.437798 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:34.498668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.498606 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:34.775820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.775795 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58"] Oct 30 13:26:34.779685 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:26:34.779658 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2978447_8e53_461f_9a9e_527f2dfc1452.slice/crio-f80ca54882f9920c9515855cc20d36767028f35200241db1ab33685a1567883c WatchSource:0}: Error finding container f80ca54882f9920c9515855cc20d36767028f35200241db1ab33685a1567883c: Status 404 returned error can't find the container with id f80ca54882f9920c9515855cc20d36767028f35200241db1ab33685a1567883c Oct 30 13:26:34.914992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:34.914912 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" event={"ID":"a2978447-8e53-461f-9a9e-527f2dfc1452","Type":"ContainerStarted","Data":"f80ca54882f9920c9515855cc20d36767028f35200241db1ab33685a1567883c"} Oct 30 13:26:35.925516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:35.925476 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" event={"ID":"a2978447-8e53-461f-9a9e-527f2dfc1452","Type":"ContainerStarted","Data":"605274dc121be8585ac4d1e64b851d4418bcc1b5c5c296fe2dc7b0719aa7feef"} Oct 30 13:26:35.965469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:35.965425 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" podStartSLOduration=1.965406612 podStartE2EDuration="1.965406612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:35.961689922 +0000 UTC m=+2102.776884344" watchObservedRunningTime="2026-10-30 13:26:35.965406612 +0000 UTC m=+2102.780601036" Oct 30 13:26:36.060946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:36.060910 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:36.923342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:36.919921 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:36.939367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:36.939303 5049 generic.go:356] "Generic (PLEG): container finished" podID="a2978447-8e53-461f-9a9e-527f2dfc1452" containerID="605274dc121be8585ac4d1e64b851d4418bcc1b5c5c296fe2dc7b0719aa7feef" exitCode=0 Oct 30 13:26:36.939367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:36.939354 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" event={"ID":"a2978447-8e53-461f-9a9e-527f2dfc1452","Type":"ContainerDied","Data":"605274dc121be8585ac4d1e64b851d4418bcc1b5c5c296fe2dc7b0719aa7feef"} Oct 30 13:26:37.657626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:37.657589 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:38.099044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.099010 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj"] Oct 30 13:26:38.153358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.153334 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:38.237827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.237744 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2978447-8e53-461f-9a9e-527f2dfc1452-kube-api-access-47qgj\" (UniqueName: \"kubernetes.io/projected/a2978447-8e53-461f-9a9e-527f2dfc1452-kube-api-access-47qgj\") pod \"a2978447-8e53-461f-9a9e-527f2dfc1452\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " Oct 30 13:26:38.244692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.244659 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-xks-ca\") pod \"a2978447-8e53-461f-9a9e-527f2dfc1452\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " Oct 30 13:26:38.249434 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.249401 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-openshift-ca\") pod \"a2978447-8e53-461f-9a9e-527f2dfc1452\" (UID: \"a2978447-8e53-461f-9a9e-527f2dfc1452\") " Oct 30 13:26:38.249808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.249782 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-openshift-ca" pod "a2978447-8e53-461f-9a9e-527f2dfc1452" (UID: "a2978447-8e53-461f-9a9e-527f2dfc1452"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:38.250148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.250126 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:38.252586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.252557 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2978447-8e53-461f-9a9e-527f2dfc1452-kube-api-access-47qgj" pod "a2978447-8e53-461f-9a9e-527f2dfc1452" (UID: "a2978447-8e53-461f-9a9e-527f2dfc1452"). InnerVolumeSpecName "kube-api-access-47qgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:38.254101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.254076 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-xks-ca" pod "a2978447-8e53-461f-9a9e-527f2dfc1452" (UID: "a2978447-8e53-461f-9a9e-527f2dfc1452"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:38.279502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.279474 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:38.352082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.352038 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qgj\" (UniqueName: \"kubernetes.io/projected/a2978447-8e53-461f-9a9e-527f2dfc1452-kube-api-access-47qgj\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:38.352082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.352079 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2978447-8e53-461f-9a9e-527f2dfc1452-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:38.449646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.449610 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:38.928221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.928186 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll"] Oct 30 13:26:38.962311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.962280 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" Oct 30 13:26:38.962514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.962316 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58" event={"ID":"a2978447-8e53-461f-9a9e-527f2dfc1452","Type":"ContainerDied","Data":"f80ca54882f9920c9515855cc20d36767028f35200241db1ab33685a1567883c"} Oct 30 13:26:38.962514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:38.962402 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f80ca54882f9920c9515855cc20d36767028f35200241db1ab33685a1567883c" Oct 30 13:26:39.008526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:39.008487 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:39.172688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:39.172611 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:39.246028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:39.245946 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" podUID="3cd54890-4ad8-4871-affd-f78510572c06" containerName="payload-processing" containerID="cri-o://0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea" gracePeriod=25 Oct 30 13:26:39.949038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:39.949001 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" podUID="3cd54890-4ad8-4871-affd-f78510572c06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:26:40.955461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:40.955367 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:41.060605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.060568 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k"] Oct 30 13:26:41.061371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.061353 5049 cpu_manager.go:414] "containerMap: removing container" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" Oct 30 13:26:41.061371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.061372 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0be57a-8d1b-4498-928c-48616b906a53" containerName="istio-proxy" Oct 30 13:26:41.061531 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.061384 5049 cpu_manager.go:414] "containerMap: removing container" podUID="a2978447-8e53-461f-9a9e-527f2dfc1452" containerName="revoke-keys" Oct 30 13:26:41.061531 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.061390 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2978447-8e53-461f-9a9e-527f2dfc1452" containerName="revoke-keys" Oct 30 13:26:41.061623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.061602 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2978447-8e53-461f-9a9e-527f2dfc1452" containerName="revoke-keys" Oct 30 13:26:41.065879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.065860 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.081074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.081048 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-9c6033\"/\"kube-root-ca.crt\"" Oct 30 13:26:41.091267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.091233 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-9c6033\"/\"openshift-service-ca.crt\"" Oct 30 13:26:41.091526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.091508 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-9c6033\"/\"default-dockercfg-xqqj2\"" Oct 30 13:26:41.091677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.091559 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-9c6033\"/\"auth-test-model-9c6033-kserve-self-signed-certs\"" Oct 30 13:26:41.097127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.097094 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k"] Oct 30 13:26:41.111367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.109438 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hddgf\" (UniqueName: \"kubernetes.io/projected/5a1441a6-7293-4524-a05f-9c0d334944d4-kube-api-access-hddgf\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.111367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.109482 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-model-cache\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.111367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.109558 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-dshm\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.111367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.109637 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-tmp-dir\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.111367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.109707 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-home\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.111367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.109788 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a1441a6-7293-4524-a05f-9c0d334944d4-tls-certs\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.166121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.166090 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:41.211512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.211432 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hddgf\" (UniqueName: \"kubernetes.io/projected/5a1441a6-7293-4524-a05f-9c0d334944d4-kube-api-access-hddgf\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.211512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.211472 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-model-cache\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.211929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.211603 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-dshm\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.211929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.211706 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-tmp-dir\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.211929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.211790 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-home\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.211929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.211859 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-model-cache\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.212177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.211947 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a1441a6-7293-4524-a05f-9c0d334944d4-tls-certs\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.212177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.212029 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-tmp-dir\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.212177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.212123 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-home\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.221112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.214955 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a1441a6-7293-4524-a05f-9c0d334944d4-tls-certs\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.221112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.215024 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-dshm\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.233205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.233176 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hddgf\" (UniqueName: \"kubernetes.io/projected/5a1441a6-7293-4524-a05f-9c0d334944d4-kube-api-access-hddgf\") pod \"auth-test-model-9c6033-kserve-b868bcd77-sbw8k\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.379983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.379705 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:41.740040 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.739929 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k"] Oct 30 13:26:41.747177 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:26:41.747142 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a1441a6_7293_4524_a05f_9c0d334944d4.slice/crio-445011d0d36738d2ebd7e3766fbcacac17e51d84a039c1e351fb86f50ac9d3d2 WatchSource:0}: Error finding container 445011d0d36738d2ebd7e3766fbcacac17e51d84a039c1e351fb86f50ac9d3d2: Status 404 returned error can't find the container with id 445011d0d36738d2ebd7e3766fbcacac17e51d84a039c1e351fb86f50ac9d3d2 Oct 30 13:26:41.988196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.988159 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" event={"ID":"5a1441a6-7293-4524-a05f-9c0d334944d4","Type":"ContainerStarted","Data":"f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1"} Oct 30 13:26:41.988727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:41.988215 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" event={"ID":"5a1441a6-7293-4524-a05f-9c0d334944d4","Type":"ContainerStarted","Data":"445011d0d36738d2ebd7e3766fbcacac17e51d84a039c1e351fb86f50ac9d3d2"} Oct 30 13:26:42.028231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:42.028097 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" podStartSLOduration=1.028078373 podStartE2EDuration="1.028078373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:42.025381342 +0000 UTC m=+2108.840575764" watchObservedRunningTime="2026-10-30 13:26:42.028078373 +0000 UTC m=+2108.843272793" Oct 30 13:26:42.379828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:42.379793 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:42.782123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:42.782008 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:43.100360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:43.100297 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" podUID="24cfc009-e002-4af1-b4a0-13b7bcf299f3" containerName="payload-processing" containerID="cri-o://c4b1b31119d9e435945f383cb60c7ea34050f02b0db8d4786798bfaa45ec2978" gracePeriod=25 Oct 30 13:26:44.785941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:44.785849 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:44.884917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:44.884873 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" podUID="24cfc009-e002-4af1-b4a0-13b7bcf299f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:26:46.686074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:46.685753 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:48.803806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:48.803775 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:49.949126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:49.949089 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" podUID="3cd54890-4ad8-4871-affd-f78510572c06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:26:50.788307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:50.788254 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:51.380365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:51.380301 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:51.394231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:51.394201 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:52.084010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:52.083980 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:26:52.637347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:52.635727 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:53.212662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.212621 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7"] Oct 30 13:26:53.218851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.218826 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.233538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.233511 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b5a369-openshift-default-dockercfg-55749\"" Oct 30 13:26:53.250636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.250598 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7"] Oct 30 13:26:53.296911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.296875 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-data\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.297078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.297020 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-podinfo\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.297133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.297086 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-envoy\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.297223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.297206 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-token\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.297288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.297270 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istiod-ca-cert\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.297376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.297363 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8w8\" (UniqueName: \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-kube-api-access-jp8w8\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.297428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.297391 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.297428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.297423 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-credential-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.297528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.297512 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-certs\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.398757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.398721 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-envoy\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.398964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.398814 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-token\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.398964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.398856 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istiod-ca-cert\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.398964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.398938 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8w8\" (UniqueName: \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-kube-api-access-jp8w8\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.399142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.398968 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.399142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.399015 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-credential-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.399142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.399051 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-certs\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.399142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.399099 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-data\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.399436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.399196 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-podinfo\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.399436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.399418 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.399606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.399583 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-certs\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.399667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.399608 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-data\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.399667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.399635 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-credential-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.399790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.399740 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istiod-ca-cert\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.401157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.401135 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-envoy\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.401701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.401683 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-podinfo\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.420178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.420145 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8w8\" (UniqueName: \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-kube-api-access-jp8w8\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.420178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.420164 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-token\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.535023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.534938 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:53.874056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.874020 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7"] Oct 30 13:26:53.878138 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:26:53.878102 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59a27ab_1b62_48fd_95ef_2e0ef55639a8.slice/crio-e36dc9a6f72f062408e990fca6282668736c9033518059263988201f18d61c47 WatchSource:0}: Error finding container e36dc9a6f72f062408e990fca6282668736c9033518059263988201f18d61c47: Status 404 returned error can't find the container with id e36dc9a6f72f062408e990fca6282668736c9033518059263988201f18d61c47 Oct 30 13:26:53.880995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.880953 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:26:53.881107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.881036 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:26:53.881107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.881083 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:26:53.958020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:53.957961 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" podUID="4489cd39-a475-496d-89f3-f38ddf93b025" containerName="main" containerID="cri-o://6211ce066ea2fa51ec816e418452ac4fa796e3394159f89647467caa655cdfa1" gracePeriod=45 Oct 30 13:26:54.095861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.095212 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" event={"ID":"f59a27ab-1b62-48fd-95ef-2e0ef55639a8","Type":"ContainerStarted","Data":"3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f"} Oct 30 13:26:54.095861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.095280 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" event={"ID":"f59a27ab-1b62-48fd-95ef-2e0ef55639a8","Type":"ContainerStarted","Data":"e36dc9a6f72f062408e990fca6282668736c9033518059263988201f18d61c47"} Oct 30 13:26:54.097791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.097753 5049 generic.go:356] "Generic (PLEG): container finished" podID="4489cd39-a475-496d-89f3-f38ddf93b025" containerID="6211ce066ea2fa51ec816e418452ac4fa796e3394159f89647467caa655cdfa1" exitCode=0 Oct 30 13:26:54.097934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.097818 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" event={"ID":"4489cd39-a475-496d-89f3-f38ddf93b025","Type":"ContainerDied","Data":"6211ce066ea2fa51ec816e418452ac4fa796e3394159f89647467caa655cdfa1"} Oct 30 13:26:54.249632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.249602 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:26:54.289451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.289399 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" podStartSLOduration=1.289384415 podStartE2EDuration="1.289384415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:54.142253498 +0000 UTC m=+2120.957447925" watchObservedRunningTime="2026-10-30 13:26:54.289384415 +0000 UTC m=+2121.104578837" Oct 30 13:26:54.417863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.417831 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-tmp-dir\") pod \"4489cd39-a475-496d-89f3-f38ddf93b025\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " Oct 30 13:26:54.418018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.417899 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-home\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-home\") pod \"4489cd39-a475-496d-89f3-f38ddf93b025\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " Oct 30 13:26:54.418018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.417969 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4489cd39-a475-496d-89f3-f38ddf93b025-kube-api-access-zgkbr\" (UniqueName: \"kubernetes.io/projected/4489cd39-a475-496d-89f3-f38ddf93b025-kube-api-access-zgkbr\") pod \"4489cd39-a475-496d-89f3-f38ddf93b025\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " Oct 30 13:26:54.418157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.418138 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4489cd39-a475-496d-89f3-f38ddf93b025-tls-certs\" (UniqueName: \"kubernetes.io/secret/4489cd39-a475-496d-89f3-f38ddf93b025-tls-certs\") pod \"4489cd39-a475-496d-89f3-f38ddf93b025\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " Oct 30 13:26:54.418679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.418248 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-model-cache\") pod \"4489cd39-a475-496d-89f3-f38ddf93b025\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " Oct 30 13:26:54.418679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.418531 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-model-cache" pod "4489cd39-a475-496d-89f3-f38ddf93b025" (UID: "4489cd39-a475-496d-89f3-f38ddf93b025"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:54.418679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.418651 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-tmp-dir" pod "4489cd39-a475-496d-89f3-f38ddf93b025" (UID: "4489cd39-a475-496d-89f3-f38ddf93b025"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:54.418927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.418603 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-home" pod "4489cd39-a475-496d-89f3-f38ddf93b025" (UID: "4489cd39-a475-496d-89f3-f38ddf93b025"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:54.418927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.418703 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-dshm\") pod \"4489cd39-a475-496d-89f3-f38ddf93b025\" (UID: \"4489cd39-a475-496d-89f3-f38ddf93b025\") " Oct 30 13:26:54.420625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.420596 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:54.420625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.420627 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:54.420811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.420641 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:54.421236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.421212 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4489cd39-a475-496d-89f3-f38ddf93b025-tls-certs" pod "4489cd39-a475-496d-89f3-f38ddf93b025" (UID: "4489cd39-a475-496d-89f3-f38ddf93b025"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:54.421446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.421417 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4489cd39-a475-496d-89f3-f38ddf93b025-kube-api-access-zgkbr" pod "4489cd39-a475-496d-89f3-f38ddf93b025" (UID: "4489cd39-a475-496d-89f3-f38ddf93b025"). InnerVolumeSpecName "kube-api-access-zgkbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:54.422627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.422606 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-dshm" pod "4489cd39-a475-496d-89f3-f38ddf93b025" (UID: "4489cd39-a475-496d-89f3-f38ddf93b025"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:54.524642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.522514 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4489cd39-a475-496d-89f3-f38ddf93b025-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:54.524642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.522557 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgkbr\" (UniqueName: \"kubernetes.io/projected/4489cd39-a475-496d-89f3-f38ddf93b025-kube-api-access-zgkbr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:54.524642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.522574 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4489cd39-a475-496d-89f3-f38ddf93b025-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:54.544619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.543436 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:26:54.544961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.544921 5049 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 13:26:54.545091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.544983 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 13:26:54.604378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.604312 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:54.884840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:54.884792 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" podUID="24cfc009-e002-4af1-b4a0-13b7bcf299f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:26:55.130346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.110868 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" Oct 30 13:26:55.133466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.115510 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll" event={"ID":"4489cd39-a475-496d-89f3-f38ddf93b025","Type":"ContainerDied","Data":"59eb34d247d2bcd66e9579e56006d6ac937618b0a2d5bb18f628777c06b2e133"} Oct 30 13:26:55.133641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.133501 5049 scope.go:122] "RemoveContainer" containerID="6211ce066ea2fa51ec816e418452ac4fa796e3394159f89647467caa655cdfa1" Oct 30 13:26:55.187011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.186983 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll"] Oct 30 13:26:55.193361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.193301 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/auth-test-model-8d82d5-kserve-5b99478c65-d68ll"] Oct 30 13:26:55.346405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.346374 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4489cd39-a475-496d-89f3-f38ddf93b025" path="/var/lib/kubelet/pods/4489cd39-a475-496d-89f3-f38ddf93b025/volumes" Oct 30 13:26:55.536207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.536099 5049 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 13:26:55.536207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.536165 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 13:26:55.836345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.834211 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:55.913530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.913312 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb"] Oct 30 13:26:55.915374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.915345 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4489cd39-a475-496d-89f3-f38ddf93b025" containerName="main" Oct 30 13:26:55.915857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.915836 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4489cd39-a475-496d-89f3-f38ddf93b025" containerName="main" Oct 30 13:26:55.916173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.916153 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4489cd39-a475-496d-89f3-f38ddf93b025" containerName="main" Oct 30 13:26:55.935682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.935657 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:55.963168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.963134 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:55.963439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.963418 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v682p\" (UniqueName: \"kubernetes.io/projected/f667f72c-036c-4324-b437-aad764d15a9e-kube-api-access-v682p\") pod \"maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:55.963725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.963708 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:55.967154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.967136 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:26:55.967280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.967172 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:26:55.989009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:55.988910 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb"] Oct 30 13:26:56.065499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:56.065461 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:56.065703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:56.065550 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:56.065703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:56.065601 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v682p\" (UniqueName: \"kubernetes.io/projected/f667f72c-036c-4324-b437-aad764d15a9e-kube-api-access-v682p\") pod \"maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:56.066229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:56.066204 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:56.067916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:56.067889 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:56.096160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:56.096099 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v682p\" (UniqueName: \"kubernetes.io/projected/f667f72c-036c-4324-b437-aad764d15a9e-kube-api-access-v682p\") pod \"maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:56.293394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:56.293358 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:56.539485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:56.539445 5049 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 13:26:56.539698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:56.539502 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 13:26:56.652122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:56.652040 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb"] Oct 30 13:26:56.686506 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:26:56.685512 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:26:57.150347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:57.150085 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:57.166226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:57.166186 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" event={"ID":"f667f72c-036c-4324-b437-aad764d15a9e","Type":"ContainerStarted","Data":"6dce3bc59ff35fbf1bc125643ff988f26ec45bec83e179c377c53c9c08c14617"} Oct 30 13:26:57.166426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:57.166257 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" event={"ID":"f667f72c-036c-4324-b437-aad764d15a9e","Type":"ContainerStarted","Data":"1f635e0a7ddbb4c99e69d303422d5c945b902ab761518716d1e285badd84d7c2"} Oct 30 13:26:57.225348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:57.217397 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" podStartSLOduration=2.2173768640000002 podStartE2EDuration="2.217376864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:57.213755678 +0000 UTC m=+2124.028950101" watchObservedRunningTime="2026-10-30 13:26:57.217376864 +0000 UTC m=+2124.032571288" Oct 30 13:26:57.535988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:57.535951 5049 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 13:26:57.536169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:57.536009 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 13:26:58.176336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:58.176297 5049 generic.go:356] "Generic (PLEG): container finished" podID="f667f72c-036c-4324-b437-aad764d15a9e" containerID="6dce3bc59ff35fbf1bc125643ff988f26ec45bec83e179c377c53c9c08c14617" exitCode=0 Oct 30 13:26:58.176740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:58.176352 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" event={"ID":"f667f72c-036c-4324-b437-aad764d15a9e","Type":"ContainerDied","Data":"6dce3bc59ff35fbf1bc125643ff988f26ec45bec83e179c377c53c9c08c14617"} Oct 30 13:26:58.537542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:58.535750 5049 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 13:26:58.537542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:58.535813 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 13:26:59.169265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.169225 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:59.455719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.455159 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:26:59.543362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.535805 5049 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 13:26:59.543362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.535866 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 13:26:59.557394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.554485 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-xks-ca\") pod \"f667f72c-036c-4324-b437-aad764d15a9e\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " Oct 30 13:26:59.557394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.554659 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f667f72c-036c-4324-b437-aad764d15a9e-kube-api-access-v682p\" (UniqueName: \"kubernetes.io/projected/f667f72c-036c-4324-b437-aad764d15a9e-kube-api-access-v682p\") pod \"f667f72c-036c-4324-b437-aad764d15a9e\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " Oct 30 13:26:59.557394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.554743 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-openshift-ca\") pod \"f667f72c-036c-4324-b437-aad764d15a9e\" (UID: \"f667f72c-036c-4324-b437-aad764d15a9e\") " Oct 30 13:26:59.557394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.555941 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-openshift-ca" pod "f667f72c-036c-4324-b437-aad764d15a9e" (UID: "f667f72c-036c-4324-b437-aad764d15a9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:59.561981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.561763 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f667f72c-036c-4324-b437-aad764d15a9e-kube-api-access-v682p" pod "f667f72c-036c-4324-b437-aad764d15a9e" (UID: "f667f72c-036c-4324-b437-aad764d15a9e"). InnerVolumeSpecName "kube-api-access-v682p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:59.566219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.566190 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-xks-ca" pod "f667f72c-036c-4324-b437-aad764d15a9e" (UID: "f667f72c-036c-4324-b437-aad764d15a9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:59.640957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.640927 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:26:59.657085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.657053 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:59.657085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.657082 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v682p\" (UniqueName: \"kubernetes.io/projected/f667f72c-036c-4324-b437-aad764d15a9e-kube-api-access-v682p\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:59.657290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.657092 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f667f72c-036c-4324-b437-aad764d15a9e-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:59.757961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.757868 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd54890-4ad8-4871-affd-f78510572c06-kube-api-access-c87sj\" (UniqueName: \"kubernetes.io/projected/3cd54890-4ad8-4871-affd-f78510572c06-kube-api-access-c87sj\") pod \"3cd54890-4ad8-4871-affd-f78510572c06\" (UID: \"3cd54890-4ad8-4871-affd-f78510572c06\") " Oct 30 13:26:59.758147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.757995 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd54890-4ad8-4871-affd-f78510572c06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd54890-4ad8-4871-affd-f78510572c06-plugins-config-volume\") pod \"3cd54890-4ad8-4871-affd-f78510572c06\" (UID: \"3cd54890-4ad8-4871-affd-f78510572c06\") " Oct 30 13:26:59.758828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.758798 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd54890-4ad8-4871-affd-f78510572c06-plugins-config-volume" pod "3cd54890-4ad8-4871-affd-f78510572c06" (UID: "3cd54890-4ad8-4871-affd-f78510572c06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:59.761255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.761228 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd54890-4ad8-4871-affd-f78510572c06-kube-api-access-c87sj" pod "3cd54890-4ad8-4871-affd-f78510572c06" (UID: "3cd54890-4ad8-4871-affd-f78510572c06"). InnerVolumeSpecName "kube-api-access-c87sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:59.859981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.859946 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c87sj\" (UniqueName: \"kubernetes.io/projected/3cd54890-4ad8-4871-affd-f78510572c06-kube-api-access-c87sj\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:26:59.859981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:26:59.859978 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd54890-4ad8-4871-affd-f78510572c06-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:00.195503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.195475 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" Oct 30 13:27:00.195716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.195505 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb" event={"ID":"f667f72c-036c-4324-b437-aad764d15a9e","Type":"ContainerDied","Data":"1f635e0a7ddbb4c99e69d303422d5c945b902ab761518716d1e285badd84d7c2"} Oct 30 13:27:00.195716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.195575 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f635e0a7ddbb4c99e69d303422d5c945b902ab761518716d1e285badd84d7c2" Oct 30 13:27:00.197231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.197202 5049 generic.go:356] "Generic (PLEG): container finished" podID="3cd54890-4ad8-4871-affd-f78510572c06" containerID="0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea" exitCode=0 Oct 30 13:27:00.197410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.197245 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" Oct 30 13:27:00.197410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.197242 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" event={"ID":"3cd54890-4ad8-4871-affd-f78510572c06","Type":"ContainerDied","Data":"0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea"} Oct 30 13:27:00.197410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.197371 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p" event={"ID":"3cd54890-4ad8-4871-affd-f78510572c06","Type":"ContainerDied","Data":"0dc4ff242857bce825398d04e3be4e59567e28024e9dadbd677d04c796081c61"} Oct 30 13:27:00.197410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.197401 5049 scope.go:122] "RemoveContainer" containerID="0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea" Oct 30 13:27:00.212760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.212736 5049 scope.go:122] "RemoveContainer" containerID="0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea" Oct 30 13:27:00.213192 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:27:00.213163 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea\": container with ID starting with 0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea not found: ID does not exist" containerID="0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea" Oct 30 13:27:00.213309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.213203 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea"} err="failed to get container status \"0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea\": rpc error: code = NotFound desc = could not find container \"0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea\": container with ID starting with 0e683f08b149350ae269cc27586bc77f0dd8d44eaeac248624f363e75ffc90ea not found: ID does not exist" Oct 30 13:27:00.262080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.262044 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p"] Oct 30 13:27:00.266855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.266828 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b5a369-6958bc65b4-xzj9p"] Oct 30 13:27:00.623549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.623514 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:27:00.624107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.624086 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:27:00.626797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.626767 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:27:00.962507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:00.962381 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:01.338054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:01.338024 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd54890-4ad8-4871-affd-f78510572c06" path="/var/lib/kubelet/pods/3cd54890-4ad8-4871-affd-f78510572c06/volumes" Oct 30 13:27:02.718341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:02.709076 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:03.236392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:03.236356 5049 generic.go:356] "Generic (PLEG): container finished" podID="24cfc009-e002-4af1-b4a0-13b7bcf299f3" containerID="c4b1b31119d9e435945f383cb60c7ea34050f02b0db8d4786798bfaa45ec2978" exitCode=0 Oct 30 13:27:03.236547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:03.236439 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" event={"ID":"24cfc009-e002-4af1-b4a0-13b7bcf299f3","Type":"ContainerDied","Data":"c4b1b31119d9e435945f383cb60c7ea34050f02b0db8d4786798bfaa45ec2978"} Oct 30 13:27:03.422568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:03.422543 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:27:03.521590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:03.521559 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24cfc009-e002-4af1-b4a0-13b7bcf299f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cfc009-e002-4af1-b4a0-13b7bcf299f3-plugins-config-volume\") pod \"24cfc009-e002-4af1-b4a0-13b7bcf299f3\" (UID: \"24cfc009-e002-4af1-b4a0-13b7bcf299f3\") " Oct 30 13:27:03.521802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:03.521782 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24cfc009-e002-4af1-b4a0-13b7bcf299f3-kube-api-access-8dzrr\" (UniqueName: \"kubernetes.io/projected/24cfc009-e002-4af1-b4a0-13b7bcf299f3-kube-api-access-8dzrr\") pod \"24cfc009-e002-4af1-b4a0-13b7bcf299f3\" (UID: \"24cfc009-e002-4af1-b4a0-13b7bcf299f3\") " Oct 30 13:27:03.522032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:03.522011 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24cfc009-e002-4af1-b4a0-13b7bcf299f3-plugins-config-volume" pod "24cfc009-e002-4af1-b4a0-13b7bcf299f3" (UID: "24cfc009-e002-4af1-b4a0-13b7bcf299f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:03.522408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:03.522386 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cfc009-e002-4af1-b4a0-13b7bcf299f3-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:03.523966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:03.523937 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24cfc009-e002-4af1-b4a0-13b7bcf299f3-kube-api-access-8dzrr" pod "24cfc009-e002-4af1-b4a0-13b7bcf299f3" (UID: "24cfc009-e002-4af1-b4a0-13b7bcf299f3"). InnerVolumeSpecName "kube-api-access-8dzrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:03.623738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:03.623702 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dzrr\" (UniqueName: \"kubernetes.io/projected/24cfc009-e002-4af1-b4a0-13b7bcf299f3-kube-api-access-8dzrr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:04.248279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:04.248239 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" event={"ID":"24cfc009-e002-4af1-b4a0-13b7bcf299f3","Type":"ContainerDied","Data":"93063e53d58f4775acbb8547098d63e63f2c079ae4b9b97911f0aaa718430bd9"} Oct 30 13:27:04.248279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:04.248272 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj" Oct 30 13:27:04.248824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:04.248309 5049 scope.go:122] "RemoveContainer" containerID="c4b1b31119d9e435945f383cb60c7ea34050f02b0db8d4786798bfaa45ec2978" Oct 30 13:27:04.256617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:04.256432 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:04.301092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:04.301055 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj"] Oct 30 13:27:04.313025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:04.312992 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5dd8cc-5c7bc8ff6-sxmvj"] Oct 30 13:27:04.888423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:04.888390 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:04.994440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:04.994408 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9"] Oct 30 13:27:05.003795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:05.003762 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b5a369-t88s9"] Oct 30 13:27:05.058247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:05.058208 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:05.217620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:05.217528 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:05.339084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:05.339049 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24cfc009-e002-4af1-b4a0-13b7bcf299f3" path="/var/lib/kubelet/pods/24cfc009-e002-4af1-b4a0-13b7bcf299f3/volumes" Oct 30 13:27:05.339750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:05.339729 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac9ac74-3545-48e1-ae41-89814ee3259b" path="/var/lib/kubelet/pods/3ac9ac74-3545-48e1-ae41-89814ee3259b/volumes" Oct 30 13:27:05.538264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:05.538177 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:06.335861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:06.335825 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p"] Oct 30 13:27:06.481254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:06.481213 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:06.639833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:06.639745 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:06.975429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:06.973287 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:09.014567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:09.014537 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:10.817647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:10.817590 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:11.336768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:11.336726 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" podUID="8d5298c6-6e5c-4513-aad1-c0c50a8b45b1" containerName="payload-pre-processing" containerID="cri-o://4e1048fc18ec13addcd78e1eb9a5a87df4c10eec65933c7a15a0fa19caac8f01" gracePeriod=25 Oct 30 13:27:12.701829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:12.701796 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:12.866666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:12.866618 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:13.076734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.076698 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7"] Oct 30 13:27:13.076971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.076951 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" containerID="cri-o://3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f" gracePeriod=30 Oct 30 13:27:13.177606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.177562 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:13.593906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.593869 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4"] Oct 30 13:27:13.594620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.594605 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3cd54890-4ad8-4871-affd-f78510572c06" containerName="payload-processing" Oct 30 13:27:13.594681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.594621 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd54890-4ad8-4871-affd-f78510572c06" containerName="payload-processing" Oct 30 13:27:13.594681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.594662 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f667f72c-036c-4324-b437-aad764d15a9e" containerName="revoke-keys" Oct 30 13:27:13.594681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.594667 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f667f72c-036c-4324-b437-aad764d15a9e" containerName="revoke-keys" Oct 30 13:27:13.594681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.594679 5049 cpu_manager.go:414] "containerMap: removing container" podUID="24cfc009-e002-4af1-b4a0-13b7bcf299f3" containerName="payload-processing" Oct 30 13:27:13.594821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.594685 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24cfc009-e002-4af1-b4a0-13b7bcf299f3" containerName="payload-processing" Oct 30 13:27:13.594821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.594781 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd54890-4ad8-4871-affd-f78510572c06" containerName="payload-processing" Oct 30 13:27:13.594821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.594795 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f667f72c-036c-4324-b437-aad764d15a9e" containerName="revoke-keys" Oct 30 13:27:13.594821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.594802 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="24cfc009-e002-4af1-b4a0-13b7bcf299f3" containerName="payload-processing" Oct 30 13:27:13.600172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.600152 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:13.625274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.625248 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:27:13.625274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.625252 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:27:13.630575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.630548 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4"] Oct 30 13:27:13.701885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.701847 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9ls\" (UniqueName: \"kubernetes.io/projected/2822c2c3-1c5b-4a4c-a45c-44727006b94a-kube-api-access-ps9ls\") pod \"maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:13.701885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.701889 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:13.702353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.701967 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:13.803960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.803928 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9ls\" (UniqueName: \"kubernetes.io/projected/2822c2c3-1c5b-4a4c-a45c-44727006b94a-kube-api-access-ps9ls\") pod \"maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:13.804152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.803974 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:13.804152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.804092 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:13.804652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.804630 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:13.806706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.806637 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:13.826868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.826841 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9ls\" (UniqueName: \"kubernetes.io/projected/2822c2c3-1c5b-4a4c-a45c-44727006b94a-kube-api-access-ps9ls\") pod \"maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:13.915442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:13.915359 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:14.274558 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.274521 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4"] Oct 30 13:27:14.276973 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:27:14.276945 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2822c2c3_1c5b_4a4c_a45c_44727006b94a.slice/crio-b5800e8d114c5af969636ec265e3a146723aa9bf8e24edfbc06ab6d41f66d837 WatchSource:0}: Error finding container b5800e8d114c5af969636ec265e3a146723aa9bf8e24edfbc06ab6d41f66d837: Status 404 returned error can't find the container with id b5800e8d114c5af969636ec265e3a146723aa9bf8e24edfbc06ab6d41f66d837 Oct 30 13:27:14.355146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.355110 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" event={"ID":"2822c2c3-1c5b-4a4c-a45c-44727006b94a","Type":"ContainerStarted","Data":"b5800e8d114c5af969636ec265e3a146723aa9bf8e24edfbc06ab6d41f66d837"} Oct 30 13:27:14.540021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.536677 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q"] Oct 30 13:27:14.551138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.551112 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.565868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.565832 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"openshift-service-ca.crt\"" Oct 30 13:27:14.567456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.567432 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q"] Oct 30 13:27:14.575818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.575792 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"sub-test-model-ff4285-kserve-self-signed-certs\"" Oct 30 13:27:14.575818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.575793 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"default-dockercfg-565c8\"" Oct 30 13:27:14.620832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.620799 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-model-cache\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.620994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.620869 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-home\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.621066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.621027 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdg5\" (UniqueName: \"kubernetes.io/projected/2458d771-9f41-48fd-9e75-78f4774c278b-kube-api-access-jwdg5\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.621066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.621061 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-tmp-dir\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.621175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.621140 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2458d771-9f41-48fd-9e75-78f4774c278b-tls-certs\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.621228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.621208 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-dshm\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.633561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.633532 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:14.665618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.665595 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"kube-root-ca.crt\"" Oct 30 13:27:14.722552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.722518 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-tmp-dir\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.722970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.722580 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2458d771-9f41-48fd-9e75-78f4774c278b-tls-certs\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.722970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.722650 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-dshm\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.722970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.722789 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-model-cache\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.722970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.722904 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-home\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.722970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.722945 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-tmp-dir\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.723281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.723060 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdg5\" (UniqueName: \"kubernetes.io/projected/2458d771-9f41-48fd-9e75-78f4774c278b-kube-api-access-jwdg5\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.723281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.723270 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-model-cache\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.723430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.723386 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-home\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.725116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.725097 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-dshm\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.725751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.725728 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2458d771-9f41-48fd-9e75-78f4774c278b-tls-certs\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.744161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.744136 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdg5\" (UniqueName: \"kubernetes.io/projected/2458d771-9f41-48fd-9e75-78f4774c278b-kube-api-access-jwdg5\") pod \"sub-test-model-ff4285-kserve-8965dffdc-lpg7q\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:14.791488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.791385 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:14.864870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:14.864832 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:15.186636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:15.186610 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q"] Oct 30 13:27:15.190104 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:27:15.190065 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2458d771_9f41_48fd_9e75_78f4774c278b.slice/crio-0574da6312e4c7f6816d04e9acaddcad7217ba36ace31274c675b026d9900b90 WatchSource:0}: Error finding container 0574da6312e4c7f6816d04e9acaddcad7217ba36ace31274c675b026d9900b90: Status 404 returned error can't find the container with id 0574da6312e4c7f6816d04e9acaddcad7217ba36ace31274c675b026d9900b90 Oct 30 13:27:15.364796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:15.364765 5049 generic.go:356] "Generic (PLEG): container finished" podID="2822c2c3-1c5b-4a4c-a45c-44727006b94a" containerID="ee57f56b9634c88d16927b16de118710c16f9e37db6030c845dce8613de3e758" exitCode=0 Oct 30 13:27:15.364966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:15.364849 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" event={"ID":"2822c2c3-1c5b-4a4c-a45c-44727006b94a","Type":"ContainerDied","Data":"ee57f56b9634c88d16927b16de118710c16f9e37db6030c845dce8613de3e758"} Oct 30 13:27:15.366620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:15.366596 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" event={"ID":"2458d771-9f41-48fd-9e75-78f4774c278b","Type":"ContainerStarted","Data":"b5e38bee59081af1c7757b20e2e94cf7ad12cc4b04db92d681f69a1b85940c25"} Oct 30 13:27:15.366723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:15.366638 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" event={"ID":"2458d771-9f41-48fd-9e75-78f4774c278b","Type":"ContainerStarted","Data":"0574da6312e4c7f6816d04e9acaddcad7217ba36ace31274c675b026d9900b90"} Oct 30 13:27:15.445455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:15.445349 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" podStartSLOduration=1.445315227 podStartE2EDuration="1.445315227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:15.439952335 +0000 UTC m=+2142.255146762" watchObservedRunningTime="2026-10-30 13:27:15.445315227 +0000 UTC m=+2142.260509652" Oct 30 13:27:15.626763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:15.626723 5049 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:27:15.626978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:15.626839 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:27:15.866025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:15.865993 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:16.515838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.515745 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:16.604685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.604658 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:16.669392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.667617 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-openshift-ca\") pod \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " Oct 30 13:27:16.669392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.667746 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2822c2c3-1c5b-4a4c-a45c-44727006b94a-kube-api-access-ps9ls\" (UniqueName: \"kubernetes.io/projected/2822c2c3-1c5b-4a4c-a45c-44727006b94a-kube-api-access-ps9ls\") pod \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " Oct 30 13:27:16.669392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.667821 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-xks-ca\") pod \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\" (UID: \"2822c2c3-1c5b-4a4c-a45c-44727006b94a\") " Oct 30 13:27:16.675771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.670581 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-openshift-ca" pod "2822c2c3-1c5b-4a4c-a45c-44727006b94a" (UID: "2822c2c3-1c5b-4a4c-a45c-44727006b94a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:16.675771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.673215 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2822c2c3-1c5b-4a4c-a45c-44727006b94a-kube-api-access-ps9ls" pod "2822c2c3-1c5b-4a4c-a45c-44727006b94a" (UID: "2822c2c3-1c5b-4a4c-a45c-44727006b94a"). InnerVolumeSpecName "kube-api-access-ps9ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:16.680278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.680225 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-xks-ca" pod "2822c2c3-1c5b-4a4c-a45c-44727006b94a" (UID: "2822c2c3-1c5b-4a4c-a45c-44727006b94a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:16.770679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.770293 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:16.770679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.770371 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2822c2c3-1c5b-4a4c-a45c-44727006b94a-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:16.770679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:16.770391 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps9ls\" (UniqueName: \"kubernetes.io/projected/2822c2c3-1c5b-4a4c-a45c-44727006b94a-kube-api-access-ps9ls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:17.126612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:17.126562 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:17.240987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:17.240950 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8"] Oct 30 13:27:17.247648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:17.247616 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd8cc-29kf8"] Oct 30 13:27:17.312430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:17.311885 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:17.338995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:17.338960 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dd8cec-22ea-4f4d-8b08-f17869fe76d5" path="/var/lib/kubelet/pods/22dd8cec-22ea-4f4d-8b08-f17869fe76d5/volumes" Oct 30 13:27:17.383865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:17.383793 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" Oct 30 13:27:17.383865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:17.383830 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4" event={"ID":"2822c2c3-1c5b-4a4c-a45c-44727006b94a","Type":"ContainerDied","Data":"b5800e8d114c5af969636ec265e3a146723aa9bf8e24edfbc06ab6d41f66d837"} Oct 30 13:27:17.384119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:17.383913 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5800e8d114c5af969636ec265e3a146723aa9bf8e24edfbc06ab6d41f66d837" Oct 30 13:27:17.786344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:17.786240 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:18.383986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.383957 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:27:18.394112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.394079 5049 generic.go:356] "Generic (PLEG): container finished" podID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerID="3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f" exitCode=0 Oct 30 13:27:18.394268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.394126 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" Oct 30 13:27:18.394268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.394154 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" event={"ID":"f59a27ab-1b62-48fd-95ef-2e0ef55639a8","Type":"ContainerDied","Data":"3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f"} Oct 30 13:27:18.394268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.394227 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7" event={"ID":"f59a27ab-1b62-48fd-95ef-2e0ef55639a8","Type":"ContainerDied","Data":"e36dc9a6f72f062408e990fca6282668736c9033518059263988201f18d61c47"} Oct 30 13:27:18.394268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.394257 5049 scope.go:122] "RemoveContainer" containerID="3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f" Oct 30 13:27:18.406361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.406311 5049 scope.go:122] "RemoveContainer" containerID="3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f" Oct 30 13:27:18.406683 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:27:18.406661 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f\": container with ID starting with 3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f not found: ID does not exist" containerID="3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f" Oct 30 13:27:18.406775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.406692 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f"} err="failed to get container status \"3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f\": rpc error: code = NotFound desc = could not find container \"3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f\": container with ID starting with 3fb4cd3e0e1bfebe61660ff6aa33804b4e8592707474aed3e4da15d87b0b340f not found: ID does not exist" Oct 30 13:27:18.500572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.500543 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-socket\") pod \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " Oct 30 13:27:18.500763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.500589 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-data\") pod \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " Oct 30 13:27:18.500763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.500628 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-podinfo\") pod \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " Oct 30 13:27:18.500763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.500699 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istiod-ca-cert\") pod \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " Oct 30 13:27:18.500961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.500847 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-data" pod "f59a27ab-1b62-48fd-95ef-2e0ef55639a8" (UID: "f59a27ab-1b62-48fd-95ef-2e0ef55639a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:18.500961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.500861 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-socket" pod "f59a27ab-1b62-48fd-95ef-2e0ef55639a8" (UID: "f59a27ab-1b62-48fd-95ef-2e0ef55639a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:18.501831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.501399 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-kube-api-access-jp8w8\" (UniqueName: \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-kube-api-access-jp8w8\") pod \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " Oct 30 13:27:18.501831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.501447 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istiod-ca-cert" pod "f59a27ab-1b62-48fd-95ef-2e0ef55639a8" (UID: "f59a27ab-1b62-48fd-95ef-2e0ef55639a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:18.501831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.501805 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-token\" (UniqueName: \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-token\") pod \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " Oct 30 13:27:18.502061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.502000 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-credential-socket\") pod \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " Oct 30 13:27:18.502061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.502031 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-certs\") pod \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " Oct 30 13:27:18.502166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.502060 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-envoy\") pod \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\" (UID: \"f59a27ab-1b62-48fd-95ef-2e0ef55639a8\") " Oct 30 13:27:18.503928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.503899 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-certs" pod "f59a27ab-1b62-48fd-95ef-2e0ef55639a8" (UID: "f59a27ab-1b62-48fd-95ef-2e0ef55639a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:18.504297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.504192 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-credential-socket" pod "f59a27ab-1b62-48fd-95ef-2e0ef55639a8" (UID: "f59a27ab-1b62-48fd-95ef-2e0ef55639a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:18.504297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.504263 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-podinfo" pod "f59a27ab-1b62-48fd-95ef-2e0ef55639a8" (UID: "f59a27ab-1b62-48fd-95ef-2e0ef55639a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:27:18.504891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.504865 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-kube-api-access-jp8w8" pod "f59a27ab-1b62-48fd-95ef-2e0ef55639a8" (UID: "f59a27ab-1b62-48fd-95ef-2e0ef55639a8"). InnerVolumeSpecName "kube-api-access-jp8w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:18.505251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.505223 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp8w8\" (UniqueName: \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-kube-api-access-jp8w8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.505404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.505255 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.505404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.505270 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.505404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.505284 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.505404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.505297 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.505404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.505310 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.505404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.505342 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.506314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.506286 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-token" pod "f59a27ab-1b62-48fd-95ef-2e0ef55639a8" (UID: "f59a27ab-1b62-48fd-95ef-2e0ef55639a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:18.506917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.506897 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-envoy" pod "f59a27ab-1b62-48fd-95ef-2e0ef55639a8" (UID: "f59a27ab-1b62-48fd-95ef-2e0ef55639a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:18.607347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.607241 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.607347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.607275 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59a27ab-1b62-48fd-95ef-2e0ef55639a8-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:18.743121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.743085 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7"] Oct 30 13:27:18.753416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.753385 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-sg5p7"] Oct 30 13:27:18.883597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:18.883503 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" podUID="8d5298c6-6e5c-4513-aad1-c0c50a8b45b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:27:19.328757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:19.328718 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:19.344150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:19.343439 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" path="/var/lib/kubelet/pods/f59a27ab-1b62-48fd-95ef-2e0ef55639a8/volumes" Oct 30 13:27:21.519276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:21.519243 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:23.438273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:23.438225 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:24.036942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:24.036905 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg"] Oct 30 13:27:24.178446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:24.178410 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:24.334672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:24.334635 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:24.865279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:24.865245 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:24.881139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:24.881112 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:25.053639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:25.053602 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:25.307006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:25.306963 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:25.467184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:25.467150 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:25.476773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:25.476748 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:27:26.283553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.283521 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:26.679054 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:27:26.679019 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:27:26.713245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.713213 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j"] Oct 30 13:27:26.714662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.714633 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" Oct 30 13:27:26.714662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.714660 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" Oct 30 13:27:26.714853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.714697 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2822c2c3-1c5b-4a4c-a45c-44727006b94a" containerName="revoke-keys" Oct 30 13:27:26.714853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.714706 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2822c2c3-1c5b-4a4c-a45c-44727006b94a" containerName="revoke-keys" Oct 30 13:27:26.714955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.714913 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59a27ab-1b62-48fd-95ef-2e0ef55639a8" containerName="istio-proxy" Oct 30 13:27:26.714955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.714936 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2822c2c3-1c5b-4a4c-a45c-44727006b94a" containerName="revoke-keys" Oct 30 13:27:26.721531 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.721342 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.738108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.738082 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c47903-openshift-default-dockercfg-wbwbt\"" Oct 30 13:27:26.742425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.742387 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-envoy\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.742674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.742637 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27db492b-5ede-4ac7-99e5-06c5bc19051f-istiod-ca-cert\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.742888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.742844 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-podinfo\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.743048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.743029 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-data\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.743239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.743209 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxz8z\" (UniqueName: \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-kube-api-access-fxz8z\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.743497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.743412 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-credential-socket\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.743733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.743709 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-certs\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.743849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.743787 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-token\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.743916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.743852 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-socket\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.759341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.759293 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j"] Oct 30 13:27:26.815820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.815782 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:26.862444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.860788 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-envoy\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.862444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.860864 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27db492b-5ede-4ac7-99e5-06c5bc19051f-istiod-ca-cert\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.862444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.860941 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-podinfo\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.862444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.860975 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-data\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.862444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.861032 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxz8z\" (UniqueName: \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-kube-api-access-fxz8z\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.862444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.861082 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-credential-socket\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.862444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.861163 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-certs\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.862444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.861253 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-token\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.862444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.861378 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-socket\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.862444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.861904 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-socket\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.863404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.862888 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-certs\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.863404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.863135 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-credential-socket\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.864270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.864237 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27db492b-5ede-4ac7-99e5-06c5bc19051f-istiod-ca-cert\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.864550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.864521 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-data\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.896340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.890677 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-envoy\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.959426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.919036 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-token\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.959426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.956026 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxz8z\" (UniqueName: \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-kube-api-access-fxz8z\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:26.999590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:26.961661 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-podinfo\") pod \"e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:27.057608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:27.057569 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:27.265162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:27.265075 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:27.390902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:27.390869 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j"] Oct 30 13:27:27.393682 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:27:27.393655 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27db492b_5ede_4ac7_99e5_06c5bc19051f.slice/crio-9c760e63b35952f2dd0bbd28884b6db4846ddb787dcc1a2506a8e35321ad8889 WatchSource:0}: Error finding container 9c760e63b35952f2dd0bbd28884b6db4846ddb787dcc1a2506a8e35321ad8889: Status 404 returned error can't find the container with id 9c760e63b35952f2dd0bbd28884b6db4846ddb787dcc1a2506a8e35321ad8889 Oct 30 13:27:27.397599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:27.397557 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:27.397731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:27.397654 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:27.397731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:27.397701 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:27.442930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:27.442898 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:27.482103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:27.482071 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" event={"ID":"27db492b-5ede-4ac7-99e5-06c5bc19051f","Type":"ContainerStarted","Data":"9c760e63b35952f2dd0bbd28884b6db4846ddb787dcc1a2506a8e35321ad8889"} Oct 30 13:27:28.218173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:28.218142 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:28.495929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:28.495832 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" event={"ID":"27db492b-5ede-4ac7-99e5-06c5bc19051f","Type":"ContainerStarted","Data":"8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b"} Oct 30 13:27:28.550335 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:28.550275 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" podStartSLOduration=2.550260482 podStartE2EDuration="2.550260482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:28.546359886 +0000 UTC m=+2155.361554310" watchObservedRunningTime="2026-10-30 13:27:28.550260482 +0000 UTC m=+2155.365454905" Oct 30 13:27:28.883625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:28.883580 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" podUID="8d5298c6-6e5c-4513-aad1-c0c50a8b45b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:27:29.038237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.038189 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" podUID="4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b" containerName="payload-processing" containerID="cri-o://1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034" gracePeriod=25 Oct 30 13:27:29.058159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.058127 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:29.065198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.065177 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:29.205860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.205823 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:29.379244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.379216 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:27:29.402312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.402270 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-plugins-config-volume\") pod \"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b\" (UID: \"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b\") " Oct 30 13:27:29.402552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.402419 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-kube-api-access-f4lkh\" (UniqueName: \"kubernetes.io/projected/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-kube-api-access-f4lkh\") pod \"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b\" (UID: \"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b\") " Oct 30 13:27:29.405460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.405214 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-plugins-config-volume" pod "4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b" (UID: "4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:29.406401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.406371 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-kube-api-access-f4lkh" pod "4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b" (UID: "4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b"). InnerVolumeSpecName "kube-api-access-f4lkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:29.504089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.504058 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:29.504089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.504086 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4lkh\" (UniqueName: \"kubernetes.io/projected/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b-kube-api-access-f4lkh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:29.505475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.505442 5049 generic.go:356] "Generic (PLEG): container finished" podID="4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b" containerID="1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034" exitCode=0 Oct 30 13:27:29.505585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.505498 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" Oct 30 13:27:29.505585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.505520 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" event={"ID":"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b","Type":"ContainerDied","Data":"1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034"} Oct 30 13:27:29.505673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.505621 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg" event={"ID":"4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b","Type":"ContainerDied","Data":"aec080aa4ae3fb149e822fa0605d2b95ee8a0294e072bc46e97f9feb19db4932"} Oct 30 13:27:29.505939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.505653 5049 scope.go:122] "RemoveContainer" containerID="1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034" Oct 30 13:27:29.506184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.506159 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:29.509392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.509375 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:27:29.526973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.526945 5049 scope.go:122] "RemoveContainer" containerID="1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034" Oct 30 13:27:29.527283 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:27:29.527261 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034\": container with ID starting with 1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034 not found: ID does not exist" containerID="1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034" Oct 30 13:27:29.527403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.527292 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034"} err="failed to get container status \"1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034\": rpc error: code = NotFound desc = could not find container \"1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034\": container with ID starting with 1e2fd87b201d336caec728255f6c1200e0dfa3a5baa56b13412afff5047e5034 not found: ID does not exist" Oct 30 13:27:29.556469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.556440 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg"] Oct 30 13:27:29.561230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:29.561202 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8467ed-5b8b988ff4-smscg"] Oct 30 13:27:30.996969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:30.996938 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:31.173444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.173411 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:31.376349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.375698 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:31.383534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.383503 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b" path="/var/lib/kubelet/pods/4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b/volumes" Oct 30 13:27:31.530903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.530868 5049 generic.go:356] "Generic (PLEG): container finished" podID="8d5298c6-6e5c-4513-aad1-c0c50a8b45b1" containerID="4e1048fc18ec13addcd78e1eb9a5a87df4c10eec65933c7a15a0fa19caac8f01" exitCode=0 Oct 30 13:27:31.531979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.531951 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" event={"ID":"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1","Type":"ContainerDied","Data":"4e1048fc18ec13addcd78e1eb9a5a87df4c10eec65933c7a15a0fa19caac8f01"} Oct 30 13:27:31.591975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.591947 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:27:31.650862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.650773 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-kube-api-access-jn299\" (UniqueName: \"kubernetes.io/projected/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-kube-api-access-jn299\") pod \"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1\" (UID: \"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1\") " Oct 30 13:27:31.651022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.650892 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-plugins-config-volume\") pod \"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1\" (UID: \"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1\") " Oct 30 13:27:31.651288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.651261 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-plugins-config-volume" pod "8d5298c6-6e5c-4513-aad1-c0c50a8b45b1" (UID: "8d5298c6-6e5c-4513-aad1-c0c50a8b45b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:31.651647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.651607 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:31.653155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.653130 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-kube-api-access-jn299" pod "8d5298c6-6e5c-4513-aad1-c0c50a8b45b1" (UID: "8d5298c6-6e5c-4513-aad1-c0c50a8b45b1"). InnerVolumeSpecName "kube-api-access-jn299". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:31.752921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:31.752881 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn299\" (UniqueName: \"kubernetes.io/projected/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1-kube-api-access-jn299\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:32.541309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:32.541275 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" event={"ID":"8d5298c6-6e5c-4513-aad1-c0c50a8b45b1","Type":"ContainerDied","Data":"af68ce268eaa82112b78ef68799e42d0d135c4c6a4dfa8a2eb4ab2fdd581e6fd"} Oct 30 13:27:32.541829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:32.541374 5049 scope.go:122] "RemoveContainer" containerID="4e1048fc18ec13addcd78e1eb9a5a87df4c10eec65933c7a15a0fa19caac8f01" Oct 30 13:27:32.541829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:32.541292 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p" Oct 30 13:27:32.591398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:32.591353 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p"] Oct 30 13:27:32.595490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:32.595461 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ad384-7fdb7bc47-9nz9p"] Oct 30 13:27:32.933470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:32.933438 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:33.016132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.016096 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f"] Oct 30 13:27:33.016836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.016820 5049 cpu_manager.go:414] "containerMap: removing container" podUID="8d5298c6-6e5c-4513-aad1-c0c50a8b45b1" containerName="payload-pre-processing" Oct 30 13:27:33.016836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.016836 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5298c6-6e5c-4513-aad1-c0c50a8b45b1" containerName="payload-pre-processing" Oct 30 13:27:33.016986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.016870 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b" containerName="payload-processing" Oct 30 13:27:33.016986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.016880 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b" containerName="payload-processing" Oct 30 13:27:33.017104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.017031 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5298c6-6e5c-4513-aad1-c0c50a8b45b1" containerName="payload-pre-processing" Oct 30 13:27:33.017104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.017044 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe53f84-bd92-4ec4-9868-a9a5a1af7b6b" containerName="payload-processing" Oct 30 13:27:33.022191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.022168 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:27:33.032550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.032522 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r"] Oct 30 13:27:33.038091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.038073 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:27:33.042225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.041861 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c47903\"" Oct 30 13:27:33.064568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.053084 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f"] Oct 30 13:27:33.064568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.056638 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c47903-dockercfg-hjd5l\"" Oct 30 13:27:33.064568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.063832 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r"] Oct 30 13:27:33.071399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.071372 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r\" (UID: \"feff1c90-5a44-4a9e-bef2-2992eeac9ee6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:27:33.071755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.071732 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqbx\" (UniqueName: \"kubernetes.io/projected/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-kube-api-access-2gqbx\") pod \"payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r\" (UID: \"feff1c90-5a44-4a9e-bef2-2992eeac9ee6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:27:33.075496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.075473 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f\" (UID: \"819b0f8d-71fd-4fed-ae5d-d619a030ffc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:27:33.075770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.075752 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s64wm\" (UniqueName: \"kubernetes.io/projected/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-kube-api-access-s64wm\") pod \"payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f\" (UID: \"819b0f8d-71fd-4fed-ae5d-d619a030ffc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:27:33.127873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.127842 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:33.192414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.181649 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f\" (UID: \"819b0f8d-71fd-4fed-ae5d-d619a030ffc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:27:33.192414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.181801 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64wm\" (UniqueName: \"kubernetes.io/projected/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-kube-api-access-s64wm\") pod \"payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f\" (UID: \"819b0f8d-71fd-4fed-ae5d-d619a030ffc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:27:33.192414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.181947 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r\" (UID: \"feff1c90-5a44-4a9e-bef2-2992eeac9ee6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:27:33.192414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.182075 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqbx\" (UniqueName: \"kubernetes.io/projected/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-kube-api-access-2gqbx\") pod \"payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r\" (UID: \"feff1c90-5a44-4a9e-bef2-2992eeac9ee6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:27:33.192414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.183123 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f\" (UID: \"819b0f8d-71fd-4fed-ae5d-d619a030ffc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:27:33.192414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.184077 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r\" (UID: \"feff1c90-5a44-4a9e-bef2-2992eeac9ee6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:27:33.259341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.259281 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqbx\" (UniqueName: \"kubernetes.io/projected/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-kube-api-access-2gqbx\") pod \"payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r\" (UID: \"feff1c90-5a44-4a9e-bef2-2992eeac9ee6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:27:33.285341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.260243 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64wm\" (UniqueName: \"kubernetes.io/projected/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-kube-api-access-s64wm\") pod \"payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f\" (UID: \"819b0f8d-71fd-4fed-ae5d-d619a030ffc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:27:33.433757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.433486 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:27:33.448398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.448274 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c47903-dockercfg-hjd5l\"" Oct 30 13:27:33.449771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.449737 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:27:33.468174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.468136 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:33.469292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.469250 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5298c6-6e5c-4513-aad1-c0c50a8b45b1" path="/var/lib/kubelet/pods/8d5298c6-6e5c-4513-aad1-c0c50a8b45b1/volumes" Oct 30 13:27:33.469771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.469753 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx"] Oct 30 13:27:33.469983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.469945 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" podUID="3557ddb3-634a-49bd-8eba-a662bcdd0bea" containerName="maas-api" containerID="cri-o://8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12" gracePeriod=30 Oct 30 13:27:33.914633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.913653 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:33.982545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:33.982518 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:27:34.010133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.010110 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r"] Oct 30 13:27:34.013576 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:27:34.013351 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeff1c90_5a44_4a9e_bef2_2992eeac9ee6.slice/crio-3ea3467d79aa077f9d0b8d1d2560a2dc2a24c772082d3d5ec698402f47c0601a WatchSource:0}: Error finding container 3ea3467d79aa077f9d0b8d1d2560a2dc2a24c772082d3d5ec698402f47c0601a: Status 404 returned error can't find the container with id 3ea3467d79aa077f9d0b8d1d2560a2dc2a24c772082d3d5ec698402f47c0601a Oct 30 13:27:34.039554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.038988 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb"] Oct 30 13:27:34.043681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.043654 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f"] Oct 30 13:27:34.047601 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:27:34.045842 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819b0f8d_71fd_4fed_ae5d_d619a030ffc5.slice/crio-8515a9727c79e6f91dbfec5b39bffc058a075a6a250ed7aa86a3cdbe51acd123 WatchSource:0}: Error finding container 8515a9727c79e6f91dbfec5b39bffc058a075a6a250ed7aa86a3cdbe51acd123: Status 404 returned error can't find the container with id 8515a9727c79e6f91dbfec5b39bffc058a075a6a250ed7aa86a3cdbe51acd123 Oct 30 13:27:34.062927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.062892 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0ad384-mp6rb"] Oct 30 13:27:34.086281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.086244 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-metrics-certs\") pod \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " Oct 30 13:27:34.093305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.093050 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-maas-api-tls\") pod \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " Oct 30 13:27:34.093305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.093112 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3557ddb3-634a-49bd-8eba-a662bcdd0bea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3557ddb3-634a-49bd-8eba-a662bcdd0bea-openshift-service-ca\") pod \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " Oct 30 13:27:34.093305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.093266 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3557ddb3-634a-49bd-8eba-a662bcdd0bea-kube-api-access-fcv99\" (UniqueName: \"kubernetes.io/projected/3557ddb3-634a-49bd-8eba-a662bcdd0bea-kube-api-access-fcv99\") pod \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\" (UID: \"3557ddb3-634a-49bd-8eba-a662bcdd0bea\") " Oct 30 13:27:34.100165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.100113 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:34.101393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.101364 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3557ddb3-634a-49bd-8eba-a662bcdd0bea-openshift-service-ca" pod "3557ddb3-634a-49bd-8eba-a662bcdd0bea" (UID: "3557ddb3-634a-49bd-8eba-a662bcdd0bea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:34.103287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.103234 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-metrics-certs" pod "3557ddb3-634a-49bd-8eba-a662bcdd0bea" (UID: "3557ddb3-634a-49bd-8eba-a662bcdd0bea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:34.103909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.103886 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-maas-api-tls" pod "3557ddb3-634a-49bd-8eba-a662bcdd0bea" (UID: "3557ddb3-634a-49bd-8eba-a662bcdd0bea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:34.104923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.104589 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3557ddb3-634a-49bd-8eba-a662bcdd0bea-kube-api-access-fcv99" pod "3557ddb3-634a-49bd-8eba-a662bcdd0bea" (UID: "3557ddb3-634a-49bd-8eba-a662bcdd0bea"). InnerVolumeSpecName "kube-api-access-fcv99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:34.195379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.195347 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcv99\" (UniqueName: \"kubernetes.io/projected/3557ddb3-634a-49bd-8eba-a662bcdd0bea-kube-api-access-fcv99\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:34.195379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.195376 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:34.195379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.195387 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3557ddb3-634a-49bd-8eba-a662bcdd0bea-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:34.195660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.195396 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3557ddb3-634a-49bd-8eba-a662bcdd0bea-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:34.563492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.563454 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j"] Oct 30 13:27:34.564632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.564605 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3557ddb3-634a-49bd-8eba-a662bcdd0bea" containerName="maas-api" Oct 30 13:27:34.564632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.564631 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3557ddb3-634a-49bd-8eba-a662bcdd0bea" containerName="maas-api" Oct 30 13:27:34.564891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.564877 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3557ddb3-634a-49bd-8eba-a662bcdd0bea" containerName="maas-api" Oct 30 13:27:34.568768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.568742 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.571697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.571674 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:34.582834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.582749 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-cda0d9\"/\"openshift-service-ca.crt\"" Oct 30 13:27:34.583277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.582901 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-cda0d9\"/\"kube-root-ca.crt\"" Oct 30 13:27:34.583445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.582805 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-cda0d9\"/\"auth-test-model-cda0d9-kserve-self-signed-certs\"" Oct 30 13:27:34.586243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.586219 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j"] Oct 30 13:27:34.590404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.590386 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-cda0d9\"/\"default-dockercfg-7r8ms\"" Oct 30 13:27:34.621051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.621013 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" event={"ID":"819b0f8d-71fd-4fed-ae5d-d619a030ffc5","Type":"ContainerStarted","Data":"2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9"} Oct 30 13:27:34.621249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.621073 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" event={"ID":"819b0f8d-71fd-4fed-ae5d-d619a030ffc5","Type":"ContainerStarted","Data":"8515a9727c79e6f91dbfec5b39bffc058a075a6a250ed7aa86a3cdbe51acd123"} Oct 30 13:27:34.621550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.621525 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:27:34.623312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.623276 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" event={"ID":"feff1c90-5a44-4a9e-bef2-2992eeac9ee6","Type":"ContainerStarted","Data":"727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2"} Oct 30 13:27:34.623312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.623345 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" event={"ID":"feff1c90-5a44-4a9e-bef2-2992eeac9ee6","Type":"ContainerStarted","Data":"3ea3467d79aa077f9d0b8d1d2560a2dc2a24c772082d3d5ec698402f47c0601a"} Oct 30 13:27:34.623565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.623418 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:27:34.625211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.625184 5049 generic.go:356] "Generic (PLEG): container finished" podID="3557ddb3-634a-49bd-8eba-a662bcdd0bea" containerID="8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12" exitCode=0 Oct 30 13:27:34.625211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.625208 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" Oct 30 13:27:34.625451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.625271 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" event={"ID":"3557ddb3-634a-49bd-8eba-a662bcdd0bea","Type":"ContainerDied","Data":"8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12"} Oct 30 13:27:34.625451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.625363 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx" event={"ID":"3557ddb3-634a-49bd-8eba-a662bcdd0bea","Type":"ContainerDied","Data":"0d02ba9bc78d4f54b745b5abe39d052deb52f6be8791a6359264b92fccdb0294"} Oct 30 13:27:34.625451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.625401 5049 scope.go:122] "RemoveContainer" containerID="8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12" Oct 30 13:27:34.640515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.640422 5049 scope.go:122] "RemoveContainer" containerID="8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12" Oct 30 13:27:34.640872 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:27:34.640849 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12\": container with ID starting with 8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12 not found: ID does not exist" containerID="8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12" Oct 30 13:27:34.641013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.640989 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12"} err="failed to get container status \"8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12\": rpc error: code = NotFound desc = could not find container \"8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12\": container with ID starting with 8b2031f6c1ad2a0c838c5b2768b55e15d5db1e8e46cfc8d8234d8e35cb47cc12 not found: ID does not exist" Oct 30 13:27:34.663225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.663161 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" podStartSLOduration=2.66314724 podStartE2EDuration="2.66314724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:34.65962297 +0000 UTC m=+2161.474817400" watchObservedRunningTime="2026-10-30 13:27:34.66314724 +0000 UTC m=+2161.478341662" Oct 30 13:27:34.698145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.698116 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx"] Oct 30 13:27:34.702499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.702474 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ad384-9f75b9b47-mgtnx"] Oct 30 13:27:34.721054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.721020 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdd9h\" (UniqueName: \"kubernetes.io/projected/d87d52c7-bb6f-417e-b214-8646eba00ff2-kube-api-access-zdd9h\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.721196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.721091 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-dshm\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.721196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.721151 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-model-cache\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.721420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.721402 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87d52c7-bb6f-417e-b214-8646eba00ff2-tls-certs\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.721488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.721476 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-tmp-dir\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.721543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.721509 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-home\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.743443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.743385 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" podStartSLOduration=2.743368463 podStartE2EDuration="2.743368463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:34.738938263 +0000 UTC m=+2161.554132687" watchObservedRunningTime="2026-10-30 13:27:34.743368463 +0000 UTC m=+2161.558562885" Oct 30 13:27:34.744483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.744460 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:34.823287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.823199 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87d52c7-bb6f-417e-b214-8646eba00ff2-tls-certs\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.823287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.823274 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-tmp-dir\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.823568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.823313 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-home\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.823568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.823397 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdd9h\" (UniqueName: \"kubernetes.io/projected/d87d52c7-bb6f-417e-b214-8646eba00ff2-kube-api-access-zdd9h\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.823568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.823436 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-dshm\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.823568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.823472 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-model-cache\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.823878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.823850 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-tmp-dir\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.823945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.823894 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-model-cache\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.824146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.824128 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-home\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.826225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.826201 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-dshm\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.826308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.826228 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87d52c7-bb6f-417e-b214-8646eba00ff2-tls-certs\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.845707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.845677 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdd9h\" (UniqueName: \"kubernetes.io/projected/d87d52c7-bb6f-417e-b214-8646eba00ff2-kube-api-access-zdd9h\") pod \"auth-test-model-cda0d9-kserve-846bb7599c-4km8j\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:34.886747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:34.886621 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:35.262802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:35.262724 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j"] Oct 30 13:27:35.266869 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:27:35.266839 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd87d52c7_bb6f_417e_b214_8646eba00ff2.slice/crio-e05ad9e40de9b9933efa41df9fe31c67c243276a3533a9fab0ef256bfcff3f5d WatchSource:0}: Error finding container e05ad9e40de9b9933efa41df9fe31c67c243276a3533a9fab0ef256bfcff3f5d: Status 404 returned error can't find the container with id e05ad9e40de9b9933efa41df9fe31c67c243276a3533a9fab0ef256bfcff3f5d Oct 30 13:27:35.353745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:35.353710 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3557ddb3-634a-49bd-8eba-a662bcdd0bea" path="/var/lib/kubelet/pods/3557ddb3-634a-49bd-8eba-a662bcdd0bea/volumes" Oct 30 13:27:35.354444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:35.354423 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f667f72c-036c-4324-b437-aad764d15a9e" path="/var/lib/kubelet/pods/f667f72c-036c-4324-b437-aad764d15a9e/volumes" Oct 30 13:27:35.635782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:35.635744 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" event={"ID":"d87d52c7-bb6f-417e-b214-8646eba00ff2","Type":"ContainerStarted","Data":"7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955"} Oct 30 13:27:35.636066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:35.636039 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" event={"ID":"d87d52c7-bb6f-417e-b214-8646eba00ff2","Type":"ContainerStarted","Data":"e05ad9e40de9b9933efa41df9fe31c67c243276a3533a9fab0ef256bfcff3f5d"} Oct 30 13:27:35.691222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:35.691156 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" podStartSLOduration=1.691136747 podStartE2EDuration="1.691136747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:35.685794654 +0000 UTC m=+2162.500989078" watchObservedRunningTime="2026-10-30 13:27:35.691136747 +0000 UTC m=+2162.506331171" Oct 30 13:27:35.887672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:35.887590 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:36.243468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:36.243380 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:38.176039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:38.176004 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:40.132452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:40.132415 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:41.230786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:41.230752 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:41.408155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:41.408119 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:41.778345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:41.768948 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:42.113855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.113815 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx"] Oct 30 13:27:42.120247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.120219 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.130772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.130054 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:42.141964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.140792 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.141964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.140980 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7x4\" (UniqueName: \"kubernetes.io/projected/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-kube-api-access-zz7x4\") pod \"maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.141964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.141066 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.142492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.142096 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx"] Oct 30 13:27:42.145011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.144988 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:27:42.145011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.144999 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:27:42.242461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.242420 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.242936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.242512 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7x4\" (UniqueName: \"kubernetes.io/projected/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-kube-api-access-zz7x4\") pod \"maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.242936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.242577 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.243128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.243104 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.244872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.244854 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.263929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.263899 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7x4\" (UniqueName: \"kubernetes.io/projected/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-kube-api-access-zz7x4\") pod \"maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.444636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.444552 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:42.768787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:42.768758 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx"] Oct 30 13:27:42.772431 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:27:42.772385 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d7339b_7a2e_4dcb_ab86_f72a706efa2d.slice/crio-9290bcd087cc3d0b94c3a67786dc2293bbb8c2df77d235551b50fd9df2837c2f WatchSource:0}: Error finding container 9290bcd087cc3d0b94c3a67786dc2293bbb8c2df77d235551b50fd9df2837c2f: Status 404 returned error can't find the container with id 9290bcd087cc3d0b94c3a67786dc2293bbb8c2df77d235551b50fd9df2837c2f Oct 30 13:27:43.718453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:43.718402 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" event={"ID":"68d7339b-7a2e-4dcb-ab86-f72a706efa2d","Type":"ContainerStarted","Data":"584a04f21a23e1076c025641d45696c87f8f8d7a16840fa509ba1f968b64f546"} Oct 30 13:27:43.718928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:43.718473 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" event={"ID":"68d7339b-7a2e-4dcb-ab86-f72a706efa2d","Type":"ContainerStarted","Data":"9290bcd087cc3d0b94c3a67786dc2293bbb8c2df77d235551b50fd9df2837c2f"} Oct 30 13:27:43.758418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:43.758372 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" podStartSLOduration=1.7583594580000002 podStartE2EDuration="1.758359458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:43.758085425 +0000 UTC m=+2170.573279848" watchObservedRunningTime="2026-10-30 13:27:43.758359458 +0000 UTC m=+2170.573553880" Oct 30 13:27:44.053769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:44.053730 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:44.593495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:44.593457 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:44.728683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:44.728648 5049 generic.go:356] "Generic (PLEG): container finished" podID="68d7339b-7a2e-4dcb-ab86-f72a706efa2d" containerID="584a04f21a23e1076c025641d45696c87f8f8d7a16840fa509ba1f968b64f546" exitCode=0 Oct 30 13:27:44.729071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:44.728723 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" event={"ID":"68d7339b-7a2e-4dcb-ab86-f72a706efa2d","Type":"ContainerDied","Data":"584a04f21a23e1076c025641d45696c87f8f8d7a16840fa509ba1f968b64f546"} Oct 30 13:27:44.887878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:44.887790 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:44.901516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:44.901490 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:45.640304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:45.640276 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:27:45.640506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:45.640357 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:27:45.760289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:45.760256 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:27:45.953400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:45.952035 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:45.999813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:45.999779 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:46.078033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.076745 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg"] Oct 30 13:27:46.080390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.078728 5049 cpu_manager.go:414] "containerMap: removing container" podUID="68d7339b-7a2e-4dcb-ab86-f72a706efa2d" containerName="revoke-keys" Oct 30 13:27:46.080390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.078755 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d7339b-7a2e-4dcb-ab86-f72a706efa2d" containerName="revoke-keys" Oct 30 13:27:46.080390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.079298 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d7339b-7a2e-4dcb-ab86-f72a706efa2d" containerName="revoke-keys" Oct 30 13:27:46.086168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.085291 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.108223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.108186 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg"] Oct 30 13:27:46.119995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.119962 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-openshift-ca\") pod \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " Oct 30 13:27:46.120170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.120147 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-xks-ca\") pod \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " Oct 30 13:27:46.120447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.120361 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-kube-api-access-zz7x4\" (UniqueName: \"kubernetes.io/projected/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-kube-api-access-zz7x4\") pod \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\" (UID: \"68d7339b-7a2e-4dcb-ab86-f72a706efa2d\") " Oct 30 13:27:46.120856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.120831 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-openshift-ca" pod "68d7339b-7a2e-4dcb-ab86-f72a706efa2d" (UID: "68d7339b-7a2e-4dcb-ab86-f72a706efa2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:46.125139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.125108 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-kube-api-access-zz7x4" pod "68d7339b-7a2e-4dcb-ab86-f72a706efa2d" (UID: "68d7339b-7a2e-4dcb-ab86-f72a706efa2d"). InnerVolumeSpecName "kube-api-access-zz7x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:46.125982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.125957 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-xks-ca" pod "68d7339b-7a2e-4dcb-ab86-f72a706efa2d" (UID: "68d7339b-7a2e-4dcb-ab86-f72a706efa2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:46.126518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.126495 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:46.126655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.126639 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz7x4\" (UniqueName: \"kubernetes.io/projected/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-kube-api-access-zz7x4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:46.126763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.126749 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68d7339b-7a2e-4dcb-ab86-f72a706efa2d-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:46.228242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.228154 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjrc\" (UniqueName: \"kubernetes.io/projected/08599d79-4e93-4a7f-8d30-d4034ab55db8-kube-api-access-kbjrc\") pod \"maas-api-revoke-sub-63997418-gz4zg\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.228422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.228302 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63997418-gz4zg\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.228422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.228413 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63997418-gz4zg\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.329999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.329966 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjrc\" (UniqueName: \"kubernetes.io/projected/08599d79-4e93-4a7f-8d30-d4034ab55db8-kube-api-access-kbjrc\") pod \"maas-api-revoke-sub-63997418-gz4zg\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.330177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.330058 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63997418-gz4zg\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.330177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.330114 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63997418-gz4zg\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.330825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.330797 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63997418-gz4zg\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.332425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.332397 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63997418-gz4zg\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.351084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.351052 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjrc\" (UniqueName: \"kubernetes.io/projected/08599d79-4e93-4a7f-8d30-d4034ab55db8-kube-api-access-kbjrc\") pod \"maas-api-revoke-sub-63997418-gz4zg\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.411906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.411873 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:46.737278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.737246 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg"] Oct 30 13:27:46.741307 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:27:46.741273 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08599d79_4e93_4a7f_8d30_d4034ab55db8.slice/crio-a7678cf849db696c1abdffb0401f0e7165d11a27396e31b86795241c24412539 WatchSource:0}: Error finding container a7678cf849db696c1abdffb0401f0e7165d11a27396e31b86795241c24412539: Status 404 returned error can't find the container with id a7678cf849db696c1abdffb0401f0e7165d11a27396e31b86795241c24412539 Oct 30 13:27:46.751023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.750998 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" Oct 30 13:27:46.751120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.751026 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx" event={"ID":"68d7339b-7a2e-4dcb-ab86-f72a706efa2d","Type":"ContainerDied","Data":"9290bcd087cc3d0b94c3a67786dc2293bbb8c2df77d235551b50fd9df2837c2f"} Oct 30 13:27:46.751120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:46.751084 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9290bcd087cc3d0b94c3a67786dc2293bbb8c2df77d235551b50fd9df2837c2f" Oct 30 13:27:47.771166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:47.771130 5049 generic.go:356] "Generic (PLEG): container finished" podID="08599d79-4e93-4a7f-8d30-d4034ab55db8" containerID="00d25c7890a4316f8f5b513498e0a73a9781a50564be7361ff757391d6d11404" exitCode=0 Oct 30 13:27:47.771655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:47.771213 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" event={"ID":"08599d79-4e93-4a7f-8d30-d4034ab55db8","Type":"ContainerDied","Data":"00d25c7890a4316f8f5b513498e0a73a9781a50564be7361ff757391d6d11404"} Oct 30 13:27:47.771655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:47.771285 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" event={"ID":"08599d79-4e93-4a7f-8d30-d4034ab55db8","Type":"ContainerStarted","Data":"a7678cf849db696c1abdffb0401f0e7165d11a27396e31b86795241c24412539"} Oct 30 13:27:47.838632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:47.838601 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:48.261770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.261740 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:48.413122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.413091 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:48.781544 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.781506 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc"] Oct 30 13:27:48.786924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.786894 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:48.801213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.801187 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-376c89-openshift-default-dockercfg-twmvq\"" Oct 30 13:27:48.816997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.816885 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc"] Oct 30 13:27:48.968943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.968906 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:48.969098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.968966 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:48.969163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.969089 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfpcv\" (UniqueName: \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-kube-api-access-gfpcv\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:48.969163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.969127 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:48.969274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.969247 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:48.969399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.969314 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:48.969399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.969396 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:48.969512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.969479 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:48.969512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.969505 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:48.989659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:48.989635 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:49.070396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070358 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.070604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070463 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.070604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070494 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.070604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070546 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.070604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070585 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.070836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070637 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpcv\" (UniqueName: \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-kube-api-access-gfpcv\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.070836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070673 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.070836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070795 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.070989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070877 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.070989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070893 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.070989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.070949 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.071255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.071233 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.071346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.071273 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.071442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.071412 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.073569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.073545 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.073726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.073649 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.093925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.093902 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpcv\" (UniqueName: \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-kube-api-access-gfpcv\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.094256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.094235 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.105424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.105398 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:49.175604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.173289 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-xks-ca\") pod \"08599d79-4e93-4a7f-8d30-d4034ab55db8\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " Oct 30 13:27:49.175604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.173377 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-openshift-ca\") pod \"08599d79-4e93-4a7f-8d30-d4034ab55db8\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " Oct 30 13:27:49.175604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.173475 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08599d79-4e93-4a7f-8d30-d4034ab55db8-kube-api-access-kbjrc\" (UniqueName: \"kubernetes.io/projected/08599d79-4e93-4a7f-8d30-d4034ab55db8-kube-api-access-kbjrc\") pod \"08599d79-4e93-4a7f-8d30-d4034ab55db8\" (UID: \"08599d79-4e93-4a7f-8d30-d4034ab55db8\") " Oct 30 13:27:49.175604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.175234 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-openshift-ca" pod "08599d79-4e93-4a7f-8d30-d4034ab55db8" (UID: "08599d79-4e93-4a7f-8d30-d4034ab55db8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:49.177879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.177856 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08599d79-4e93-4a7f-8d30-d4034ab55db8-kube-api-access-kbjrc" pod "08599d79-4e93-4a7f-8d30-d4034ab55db8" (UID: "08599d79-4e93-4a7f-8d30-d4034ab55db8"). InnerVolumeSpecName "kube-api-access-kbjrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:49.180099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.179932 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-xks-ca" pod "08599d79-4e93-4a7f-8d30-d4034ab55db8" (UID: "08599d79-4e93-4a7f-8d30-d4034ab55db8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:49.276412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.276375 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:49.276412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.276410 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08599d79-4e93-4a7f-8d30-d4034ab55db8-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:49.276629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.276425 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbjrc\" (UniqueName: \"kubernetes.io/projected/08599d79-4e93-4a7f-8d30-d4034ab55db8-kube-api-access-kbjrc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:27:49.398795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.398647 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:49.445613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.445580 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc"] Oct 30 13:27:49.445972 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:27:49.445946 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54e3b09_d1da_44b3_9f72_0371d05e26c4.slice/crio-930c864d92adeb18d7164523b9b3c0cb0d23f81dae560956a3851bf8150b62d0 WatchSource:0}: Error finding container 930c864d92adeb18d7164523b9b3c0cb0d23f81dae560956a3851bf8150b62d0: Status 404 returned error can't find the container with id 930c864d92adeb18d7164523b9b3c0cb0d23f81dae560956a3851bf8150b62d0 Oct 30 13:27:49.449908 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.449869 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:49.450014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.449956 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:49.450014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.449998 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:49.643742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.643642 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:49.795020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.794996 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" Oct 30 13:27:49.795517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.795021 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg" event={"ID":"08599d79-4e93-4a7f-8d30-d4034ab55db8","Type":"ContainerDied","Data":"a7678cf849db696c1abdffb0401f0e7165d11a27396e31b86795241c24412539"} Oct 30 13:27:49.795517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.795076 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7678cf849db696c1abdffb0401f0e7165d11a27396e31b86795241c24412539" Oct 30 13:27:49.797193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.797129 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" event={"ID":"e54e3b09-d1da-44b3-9f72-0371d05e26c4","Type":"ContainerStarted","Data":"aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c"} Oct 30 13:27:49.797193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.797190 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" event={"ID":"e54e3b09-d1da-44b3-9f72-0371d05e26c4","Type":"ContainerStarted","Data":"930c864d92adeb18d7164523b9b3c0cb0d23f81dae560956a3851bf8150b62d0"} Oct 30 13:27:49.851979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:49.851928 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" podStartSLOduration=1.851914658 podStartE2EDuration="1.851914658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:49.838315451 +0000 UTC m=+2176.653509874" watchObservedRunningTime="2026-10-30 13:27:49.851914658 +0000 UTC m=+2176.667109081" Oct 30 13:27:50.111708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:50.111626 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:50.125108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:50.125070 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:27:50.125412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:50.125383 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 13:27:51.115254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:51.115214 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:27:51.115868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:51.115832 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 13:27:51.534854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:51.534741 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:52.109757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:52.109719 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:27:52.110065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:52.110035 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 13:27:52.692945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:52.692907 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:52.853084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:52.853042 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:53.117758 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:53.117714 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:27:53.118069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:53.118040 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 13:27:53.266353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:53.264399 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:54.119722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.119686 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:27:54.120314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.120283 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 13:27:54.143782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.143746 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g"] Oct 30 13:27:54.145185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.145157 5049 cpu_manager.go:414] "containerMap: removing container" podUID="08599d79-4e93-4a7f-8d30-d4034ab55db8" containerName="revoke-keys" Oct 30 13:27:54.145377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.145361 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08599d79-4e93-4a7f-8d30-d4034ab55db8" containerName="revoke-keys" Oct 30 13:27:54.145703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.145689 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="08599d79-4e93-4a7f-8d30-d4034ab55db8" containerName="revoke-keys" Oct 30 13:27:54.153356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.153304 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.155491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.155462 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g"] Oct 30 13:27:54.166907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.166877 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-9c6033-openshift-default-dockercfg-5nxq8\"" Oct 30 13:27:54.204225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.204189 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-data\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.204444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.204241 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-podinfo\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.204444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.204411 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-credential-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.204549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.204472 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-token\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.204587 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.204563 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-envoy\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.204685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.204667 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-certs\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.204770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.204754 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istiod-ca-cert\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.204904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.204879 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6v74\" (UniqueName: \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-kube-api-access-c6v74\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.205003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.204986 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.311346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.310258 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.311346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.310405 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-data\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.311346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.310436 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-podinfo\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.311346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.310525 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-credential-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.311346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.310576 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-token\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.311346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.310626 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-envoy\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.311346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.310709 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-certs\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.311346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.310796 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istiod-ca-cert\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.311346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.310880 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6v74\" (UniqueName: \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-kube-api-access-c6v74\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.311980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.311717 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.312040 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.311975 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-data\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.312951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.312926 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-credential-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.324399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.313361 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-certs\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.324590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.314185 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istiod-ca-cert\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.347360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.345282 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-envoy\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.347360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.345737 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-token\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.348779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.348738 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-podinfo\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.349151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.349127 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6v74\" (UniqueName: \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-kube-api-access-c6v74\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.471595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.471499 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:54.848193 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:27:54.848152 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee84f7f1_7d54_4354_bd2b_872e0cb59b75.slice/crio-79ea4d9b245655a1e2dd9236cebd1d93f274cb52b3c57d7000d6bcbf807dc1ec WatchSource:0}: Error finding container 79ea4d9b245655a1e2dd9236cebd1d93f274cb52b3c57d7000d6bcbf807dc1ec: Status 404 returned error can't find the container with id 79ea4d9b245655a1e2dd9236cebd1d93f274cb52b3c57d7000d6bcbf807dc1ec Oct 30 13:27:54.848984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.848960 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g"] Oct 30 13:27:54.852149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.852111 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:54.852240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.852209 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:54.852282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.852253 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:27:54.874582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:54.874549 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" event={"ID":"ee84f7f1-7d54-4354-bd2b-872e0cb59b75","Type":"ContainerStarted","Data":"79ea4d9b245655a1e2dd9236cebd1d93f274cb52b3c57d7000d6bcbf807dc1ec"} Oct 30 13:27:55.111402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:55.109746 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 13:27:55.111402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:55.109822 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 13:27:55.200573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:55.200538 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:55.889615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:55.889559 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" event={"ID":"ee84f7f1-7d54-4354-bd2b-872e0cb59b75","Type":"ContainerStarted","Data":"9ec0c406c43ea80acbc725bb189e6d2ff05182b6517ee5304975a94dceb9d477"} Oct 30 13:27:55.953857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:55.953774 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" podStartSLOduration=1.9537522520000001 podStartE2EDuration="1.953752252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:55.948974972 +0000 UTC m=+2182.764169399" watchObservedRunningTime="2026-10-30 13:27:55.953752252 +0000 UTC m=+2182.768946677" Oct 30 13:27:56.157886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:56.156886 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:56.158392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:56.158365 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:56.160753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:56.160730 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:27:56.359585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:56.359537 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:56.471741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:56.471658 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:27:56.473471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:56.473446 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:27:56.473596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:56.473491 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 13:27:56.689082 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:27:56.689046 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:27:57.073268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:57.073233 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:57.472015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:57.471933 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:27:57.472015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:57.471997 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 13:27:58.472368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:58.472333 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:27:58.472736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:58.472395 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 13:27:58.879454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:58.878359 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:59.471962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:59.471921 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:27:59.472188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:27:59.471999 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 13:28:00.450109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:00.450066 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:00.471901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:00.471868 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:28:00.472042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:00.471932 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 13:28:01.138441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:01.138383 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:01.250379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:01.250343 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4"] Oct 30 13:28:01.259168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:01.259139 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8467ed-gr9m4"] Oct 30 13:28:01.294517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:01.294486 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:01.349536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:01.349502 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2822c2c3-1c5b-4a4c-a45c-44727006b94a" path="/var/lib/kubelet/pods/2822c2c3-1c5b-4a4c-a45c-44727006b94a/volumes" Oct 30 13:28:01.445730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:01.445639 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:01.472227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:01.472186 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:28:01.472684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:01.472267 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 13:28:01.755846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:01.755745 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:02.478545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:02.478517 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:28:02.479035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:02.479021 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:28:02.479692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:02.479666 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:28:03.232911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:03.232873 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:04.822437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:04.822407 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:05.000109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.000071 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:05.335905 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.335868 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:05.754495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.754409 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs"] Oct 30 13:28:05.761106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.761083 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:05.781843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.778581 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs"] Oct 30 13:28:05.783102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.783075 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:28:05.783277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.783202 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:28:05.792220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.791719 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6s22\" (UniqueName: \"kubernetes.io/projected/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-kube-api-access-v6s22\") pod \"maas-api-revoke-sub-bf6e4f8f-m65bs\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:05.792220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.791839 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6e4f8f-m65bs\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:05.792220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.792022 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6e4f8f-m65bs\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:05.894163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.894126 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6s22\" (UniqueName: \"kubernetes.io/projected/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-kube-api-access-v6s22\") pod \"maas-api-revoke-sub-bf6e4f8f-m65bs\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:05.894602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.894189 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6e4f8f-m65bs\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:05.894602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.894287 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6e4f8f-m65bs\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:05.894898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.894866 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6e4f8f-m65bs\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:05.896702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.896677 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6e4f8f-m65bs\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:05.915596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:05.915564 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6s22\" (UniqueName: \"kubernetes.io/projected/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-kube-api-access-v6s22\") pod \"maas-api-revoke-sub-bf6e4f8f-m65bs\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:06.078059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:06.078027 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:06.395588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:06.395553 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs"] Oct 30 13:28:06.398528 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:28:06.398495 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfb32b0_b999_4f64_ba2f_efa5eb135bd4.slice/crio-02208578a82f94fa42cdd0542c5cdcd0cd903b062a63ce014619d4a67ae41bd9 WatchSource:0}: Error finding container 02208578a82f94fa42cdd0542c5cdcd0cd903b062a63ce014619d4a67ae41bd9: Status 404 returned error can't find the container with id 02208578a82f94fa42cdd0542c5cdcd0cd903b062a63ce014619d4a67ae41bd9 Oct 30 13:28:07.014034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:07.013945 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:07.092675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:07.092639 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" event={"ID":"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4","Type":"ContainerStarted","Data":"9ba827d5c6d365d01eb96b6d15af0f9216efbba5dafdbba612974b6e7d66b7e3"} Oct 30 13:28:07.092881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:07.092695 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" event={"ID":"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4","Type":"ContainerStarted","Data":"02208578a82f94fa42cdd0542c5cdcd0cd903b062a63ce014619d4a67ae41bd9"} Oct 30 13:28:07.136731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:07.136666 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" podStartSLOduration=2.136644123 podStartE2EDuration="2.136644123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:07.129756556 +0000 UTC m=+2193.944950980" watchObservedRunningTime="2026-10-30 13:28:07.136644123 +0000 UTC m=+2193.951838548" Oct 30 13:28:08.106180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:08.106140 5049 generic.go:356] "Generic (PLEG): container finished" podID="8cfb32b0-b999-4f64-ba2f-efa5eb135bd4" containerID="9ba827d5c6d365d01eb96b6d15af0f9216efbba5dafdbba612974b6e7d66b7e3" exitCode=0 Oct 30 13:28:08.106664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:08.106189 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" event={"ID":"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4","Type":"ContainerDied","Data":"9ba827d5c6d365d01eb96b6d15af0f9216efbba5dafdbba612974b6e7d66b7e3"} Oct 30 13:28:08.460783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:08.460697 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:08.612278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:08.612244 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:08.805556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:08.805519 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:08.968247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:08.968213 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:09.303973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:09.303940 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:09.372160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:09.372081 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-kube-api-access-v6s22\" (UniqueName: \"kubernetes.io/projected/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-kube-api-access-v6s22\") pod \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " Oct 30 13:28:09.372160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:09.372123 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-openshift-ca\") pod \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " Oct 30 13:28:09.372453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:09.372349 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-xks-ca\") pod \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\" (UID: \"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4\") " Oct 30 13:28:09.372575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:09.372553 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-openshift-ca" pod "8cfb32b0-b999-4f64-ba2f-efa5eb135bd4" (UID: "8cfb32b0-b999-4f64-ba2f-efa5eb135bd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:09.373030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:09.373008 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:09.374402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:09.374380 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-xks-ca" pod "8cfb32b0-b999-4f64-ba2f-efa5eb135bd4" (UID: "8cfb32b0-b999-4f64-ba2f-efa5eb135bd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:09.374583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:09.374558 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-kube-api-access-v6s22" pod "8cfb32b0-b999-4f64-ba2f-efa5eb135bd4" (UID: "8cfb32b0-b999-4f64-ba2f-efa5eb135bd4"). InnerVolumeSpecName "kube-api-access-v6s22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:09.475208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:09.475173 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:09.475208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:09.475208 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6s22\" (UniqueName: \"kubernetes.io/projected/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4-kube-api-access-v6s22\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:10.131334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:10.131290 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" Oct 30 13:28:10.131575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:10.131307 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs" event={"ID":"8cfb32b0-b999-4f64-ba2f-efa5eb135bd4","Type":"ContainerDied","Data":"02208578a82f94fa42cdd0542c5cdcd0cd903b062a63ce014619d4a67ae41bd9"} Oct 30 13:28:10.131575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:10.131393 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02208578a82f94fa42cdd0542c5cdcd0cd903b062a63ce014619d4a67ae41bd9" Oct 30 13:28:10.751342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:10.750113 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:12.540606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:12.540576 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:12.697924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:12.697889 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:12.861885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:12.861848 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:14.505396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:14.503458 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:16.470757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:16.470716 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:18.364275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:18.364237 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:20.223476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:20.223440 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:22.193291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.193258 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:22.405785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.405746 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:22.544381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.544348 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b"] Oct 30 13:28:22.545600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.545573 5049 cpu_manager.go:414] "containerMap: removing container" podUID="8cfb32b0-b999-4f64-ba2f-efa5eb135bd4" containerName="revoke-keys" Oct 30 13:28:22.545600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.545599 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfb32b0-b999-4f64-ba2f-efa5eb135bd4" containerName="revoke-keys" Oct 30 13:28:22.546283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.546243 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfb32b0-b999-4f64-ba2f-efa5eb135bd4" containerName="revoke-keys" Oct 30 13:28:22.552347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.552308 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:22.574361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.574314 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b"] Oct 30 13:28:22.575441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.574701 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:28:22.575441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.575367 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:28:22.586958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.586851 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctvf\" (UniqueName: \"kubernetes.io/projected/52653f4c-bcb3-4095-9cda-0ff663771097-kube-api-access-cctvf\") pod \"maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:22.586958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.586952 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:22.587181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.587062 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:22.689048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.689018 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:22.689243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.689134 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:22.689431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.689412 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctvf\" (UniqueName: \"kubernetes.io/projected/52653f4c-bcb3-4095-9cda-0ff663771097-kube-api-access-cctvf\") pod \"maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:22.689852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.689829 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:22.691758 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.691735 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:22.711449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.711422 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctvf\" (UniqueName: \"kubernetes.io/projected/52653f4c-bcb3-4095-9cda-0ff663771097-kube-api-access-cctvf\") pod \"maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:22.872527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:22.872431 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:23.195368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:23.192707 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b"] Oct 30 13:28:23.195368 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:28:23.195022 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52653f4c_bcb3_4095_9cda_0ff663771097.slice/crio-1712fc969880d0d60cdf346da7f69c1bfdf72f72722394a36767939681d33bd4 WatchSource:0}: Error finding container 1712fc969880d0d60cdf346da7f69c1bfdf72f72722394a36767939681d33bd4: Status 404 returned error can't find the container with id 1712fc969880d0d60cdf346da7f69c1bfdf72f72722394a36767939681d33bd4 Oct 30 13:28:23.255895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:23.255862 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" event={"ID":"52653f4c-bcb3-4095-9cda-0ff663771097","Type":"ContainerStarted","Data":"1712fc969880d0d60cdf346da7f69c1bfdf72f72722394a36767939681d33bd4"} Oct 30 13:28:23.693603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:23.693561 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g"] Oct 30 13:28:23.694311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:23.694005 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" podUID="e11b2731-225a-4b4a-9b36-76d4a44e2239" containerName="maas-api" containerID="cri-o://d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a" gracePeriod=30 Oct 30 13:28:23.805888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:23.805854 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:24.088971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.088946 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:28:24.124342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.124248 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e11b2731-225a-4b4a-9b36-76d4a44e2239-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11b2731-225a-4b4a-9b36-76d4a44e2239-openshift-service-ca\") pod \"e11b2731-225a-4b4a-9b36-76d4a44e2239\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " Oct 30 13:28:24.124508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.124400 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11b2731-225a-4b4a-9b36-76d4a44e2239-kube-api-access-hbcd2\" (UniqueName: \"kubernetes.io/projected/e11b2731-225a-4b4a-9b36-76d4a44e2239-kube-api-access-hbcd2\") pod \"e11b2731-225a-4b4a-9b36-76d4a44e2239\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " Oct 30 13:28:24.124574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.124542 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-metrics-certs\") pod \"e11b2731-225a-4b4a-9b36-76d4a44e2239\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " Oct 30 13:28:24.124633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.124589 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11b2731-225a-4b4a-9b36-76d4a44e2239-openshift-service-ca" pod "e11b2731-225a-4b4a-9b36-76d4a44e2239" (UID: "e11b2731-225a-4b4a-9b36-76d4a44e2239"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:24.124701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.124686 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-maas-api-tls\") pod \"e11b2731-225a-4b4a-9b36-76d4a44e2239\" (UID: \"e11b2731-225a-4b4a-9b36-76d4a44e2239\") " Oct 30 13:28:24.125278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.125258 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11b2731-225a-4b4a-9b36-76d4a44e2239-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:24.126704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.126680 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-metrics-certs" pod "e11b2731-225a-4b4a-9b36-76d4a44e2239" (UID: "e11b2731-225a-4b4a-9b36-76d4a44e2239"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:24.127028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.127004 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-maas-api-tls" pod "e11b2731-225a-4b4a-9b36-76d4a44e2239" (UID: "e11b2731-225a-4b4a-9b36-76d4a44e2239"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:24.127165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.127144 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11b2731-225a-4b4a-9b36-76d4a44e2239-kube-api-access-hbcd2" pod "e11b2731-225a-4b4a-9b36-76d4a44e2239" (UID: "e11b2731-225a-4b4a-9b36-76d4a44e2239"). InnerVolumeSpecName "kube-api-access-hbcd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:24.227235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.227202 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbcd2\" (UniqueName: \"kubernetes.io/projected/e11b2731-225a-4b4a-9b36-76d4a44e2239-kube-api-access-hbcd2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:24.227235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.227236 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:24.227722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.227250 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b2731-225a-4b4a-9b36-76d4a44e2239-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:24.274348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.271657 5049 generic.go:356] "Generic (PLEG): container finished" podID="e11b2731-225a-4b4a-9b36-76d4a44e2239" containerID="d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a" exitCode=0 Oct 30 13:28:24.274348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.271765 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" event={"ID":"e11b2731-225a-4b4a-9b36-76d4a44e2239","Type":"ContainerDied","Data":"d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a"} Oct 30 13:28:24.274348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.271812 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" event={"ID":"e11b2731-225a-4b4a-9b36-76d4a44e2239","Type":"ContainerDied","Data":"8f29b564fd88a7e22ab1145784d3f0390b7741a364dfaa27c05acaec04234966"} Oct 30 13:28:24.274348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.271839 5049 scope.go:122] "RemoveContainer" containerID="d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a" Oct 30 13:28:24.274348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.271987 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g" Oct 30 13:28:24.299681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.285156 5049 generic.go:356] "Generic (PLEG): container finished" podID="52653f4c-bcb3-4095-9cda-0ff663771097" containerID="b7375fce36f2c3a0017f760042dadf7b3bb27ed0c095456f330e8ca481d21fdd" exitCode=0 Oct 30 13:28:24.299681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.285441 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" event={"ID":"52653f4c-bcb3-4095-9cda-0ff663771097","Type":"ContainerDied","Data":"b7375fce36f2c3a0017f760042dadf7b3bb27ed0c095456f330e8ca481d21fdd"} Oct 30 13:28:24.350343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.347360 5049 scope.go:122] "RemoveContainer" containerID="d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a" Oct 30 13:28:24.350343 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:28:24.348499 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a\": container with ID starting with d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a not found: ID does not exist" containerID="d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a" Oct 30 13:28:24.350343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.348536 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a"} err="failed to get container status \"d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a\": rpc error: code = NotFound desc = could not find container \"d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a\": container with ID starting with d9397ae77e5ed1e37601e63e03b4f74d8aa67e1a3564df254860e0dcfa2fb63a not found: ID does not exist" Oct 30 13:28:24.369218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.369172 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g"] Oct 30 13:28:24.376766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.376637 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09e4c0-67558bd879-bz95g"] Oct 30 13:28:24.517338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.517294 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:24.618972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.618932 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx"] Oct 30 13:28:24.623842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.623819 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09e4c0-fxzmx"] Oct 30 13:28:24.671998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.671914 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:24.820352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:24.820283 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:25.135853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.135818 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:25.340127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.340081 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d7339b-7a2e-4dcb-ab86-f72a706efa2d" path="/var/lib/kubelet/pods/68d7339b-7a2e-4dcb-ab86-f72a706efa2d/volumes" Oct 30 13:28:25.340821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.340798 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e11b2731-225a-4b4a-9b36-76d4a44e2239" path="/var/lib/kubelet/pods/e11b2731-225a-4b4a-9b36-76d4a44e2239/volumes" Oct 30 13:28:25.489017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.488990 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:25.571257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.571173 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-openshift-ca\") pod \"52653f4c-bcb3-4095-9cda-0ff663771097\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " Oct 30 13:28:25.571465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.571445 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52653f4c-bcb3-4095-9cda-0ff663771097-kube-api-access-cctvf\" (UniqueName: \"kubernetes.io/projected/52653f4c-bcb3-4095-9cda-0ff663771097-kube-api-access-cctvf\") pod \"52653f4c-bcb3-4095-9cda-0ff663771097\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " Oct 30 13:28:25.571600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.571571 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-openshift-ca" pod "52653f4c-bcb3-4095-9cda-0ff663771097" (UID: "52653f4c-bcb3-4095-9cda-0ff663771097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:25.571697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.571682 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-xks-ca\") pod \"52653f4c-bcb3-4095-9cda-0ff663771097\" (UID: \"52653f4c-bcb3-4095-9cda-0ff663771097\") " Oct 30 13:28:25.572245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.572225 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:25.573679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.573652 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-xks-ca" pod "52653f4c-bcb3-4095-9cda-0ff663771097" (UID: "52653f4c-bcb3-4095-9cda-0ff663771097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:25.573883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.573858 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52653f4c-bcb3-4095-9cda-0ff663771097-kube-api-access-cctvf" pod "52653f4c-bcb3-4095-9cda-0ff663771097" (UID: "52653f4c-bcb3-4095-9cda-0ff663771097"). InnerVolumeSpecName "kube-api-access-cctvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:25.673843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.673808 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52653f4c-bcb3-4095-9cda-0ff663771097-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:25.673843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:25.673837 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cctvf\" (UniqueName: \"kubernetes.io/projected/52653f4c-bcb3-4095-9cda-0ff663771097-kube-api-access-cctvf\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:26.305242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:26.305215 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" Oct 30 13:28:26.305474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:26.305246 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b" event={"ID":"52653f4c-bcb3-4095-9cda-0ff663771097","Type":"ContainerDied","Data":"1712fc969880d0d60cdf346da7f69c1bfdf72f72722394a36767939681d33bd4"} Oct 30 13:28:26.305474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:26.305292 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1712fc969880d0d60cdf346da7f69c1bfdf72f72722394a36767939681d33bd4" Oct 30 13:28:26.694237 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:28:26.694138 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:28:26.697139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:26.697113 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:27.394341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.373002 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr"] Oct 30 13:28:27.504382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.503245 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:27.585071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.585039 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2"] Oct 30 13:28:27.586175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.586146 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e11b2731-225a-4b4a-9b36-76d4a44e2239" containerName="maas-api" Oct 30 13:28:27.586338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.586175 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11b2731-225a-4b4a-9b36-76d4a44e2239" containerName="maas-api" Oct 30 13:28:27.586338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.586251 5049 cpu_manager.go:414] "containerMap: removing container" podUID="52653f4c-bcb3-4095-9cda-0ff663771097" containerName="revoke-keys" Oct 30 13:28:27.586338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.586260 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52653f4c-bcb3-4095-9cda-0ff663771097" containerName="revoke-keys" Oct 30 13:28:27.586544 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.586528 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11b2731-225a-4b4a-9b36-76d4a44e2239" containerName="maas-api" Oct 30 13:28:27.586609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.586549 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="52653f4c-bcb3-4095-9cda-0ff663771097" containerName="revoke-keys" Oct 30 13:28:27.591456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.591431 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:27.607079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.606560 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:28:27.609887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.609858 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:28:27.621430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.621399 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2"] Oct 30 13:28:27.703910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.703829 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:27.704277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.704064 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkgk\" (UniqueName: \"kubernetes.io/projected/3f1089e5-8890-4d38-b17e-78b2b651e392-kube-api-access-cdkgk\") pod \"maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:27.704277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.704106 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:27.805473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.805436 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:27.805685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.805605 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkgk\" (UniqueName: \"kubernetes.io/projected/3f1089e5-8890-4d38-b17e-78b2b651e392-kube-api-access-cdkgk\") pod \"maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:27.805685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.805639 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:27.806157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.806130 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:27.808196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.808168 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:27.827444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.827414 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkgk\" (UniqueName: \"kubernetes.io/projected/3f1089e5-8890-4d38-b17e-78b2b651e392-kube-api-access-cdkgk\") pod \"maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:27.944373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:27.944309 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:28.267551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:28.267522 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2"] Oct 30 13:28:28.270522 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:28:28.270496 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1089e5_8890_4d38_b17e_78b2b651e392.slice/crio-1da851b3b56796b5df83e773031eebfc05d90d7f983d0b1e5b7bb3e6ae06a4b8 WatchSource:0}: Error finding container 1da851b3b56796b5df83e773031eebfc05d90d7f983d0b1e5b7bb3e6ae06a4b8: Status 404 returned error can't find the container with id 1da851b3b56796b5df83e773031eebfc05d90d7f983d0b1e5b7bb3e6ae06a4b8 Oct 30 13:28:28.329240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:28.329205 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" event={"ID":"3f1089e5-8890-4d38-b17e-78b2b651e392","Type":"ContainerStarted","Data":"1da851b3b56796b5df83e773031eebfc05d90d7f983d0b1e5b7bb3e6ae06a4b8"} Oct 30 13:28:28.340213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:28.340188 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:28.988731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:28.988592 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:29.346446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:29.346412 5049 generic.go:356] "Generic (PLEG): container finished" podID="3f1089e5-8890-4d38-b17e-78b2b651e392" containerID="770386b00be09d5849427ee44f7bf29ae58266d42eeb424a9e5ca6b29a27c498" exitCode=0 Oct 30 13:28:29.346610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:29.346571 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" event={"ID":"3f1089e5-8890-4d38-b17e-78b2b651e392","Type":"ContainerDied","Data":"770386b00be09d5849427ee44f7bf29ae58266d42eeb424a9e5ca6b29a27c498"} Oct 30 13:28:30.543678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.543649 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:30.653088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.653002 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-openshift-ca\") pod \"3f1089e5-8890-4d38-b17e-78b2b651e392\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " Oct 30 13:28:30.653088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.653077 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1089e5-8890-4d38-b17e-78b2b651e392-kube-api-access-cdkgk\" (UniqueName: \"kubernetes.io/projected/3f1089e5-8890-4d38-b17e-78b2b651e392-kube-api-access-cdkgk\") pod \"3f1089e5-8890-4d38-b17e-78b2b651e392\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " Oct 30 13:28:30.653348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.653270 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-xks-ca\") pod \"3f1089e5-8890-4d38-b17e-78b2b651e392\" (UID: \"3f1089e5-8890-4d38-b17e-78b2b651e392\") " Oct 30 13:28:30.653526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.653497 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-openshift-ca" pod "3f1089e5-8890-4d38-b17e-78b2b651e392" (UID: "3f1089e5-8890-4d38-b17e-78b2b651e392"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:30.653949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.653927 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:30.655837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.655809 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-xks-ca" pod "3f1089e5-8890-4d38-b17e-78b2b651e392" (UID: "3f1089e5-8890-4d38-b17e-78b2b651e392"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:30.655926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.655860 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1089e5-8890-4d38-b17e-78b2b651e392-kube-api-access-cdkgk" pod "3f1089e5-8890-4d38-b17e-78b2b651e392" (UID: "3f1089e5-8890-4d38-b17e-78b2b651e392"). InnerVolumeSpecName "kube-api-access-cdkgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:30.755213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.755181 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdkgk\" (UniqueName: \"kubernetes.io/projected/3f1089e5-8890-4d38-b17e-78b2b651e392-kube-api-access-cdkgk\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:30.755213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.755211 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1089e5-8890-4d38-b17e-78b2b651e392-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:30.779106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:30.779071 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:31.364931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:31.364894 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" event={"ID":"3f1089e5-8890-4d38-b17e-78b2b651e392","Type":"ContainerDied","Data":"1da851b3b56796b5df83e773031eebfc05d90d7f983d0b1e5b7bb3e6ae06a4b8"} Oct 30 13:28:31.365094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:31.364953 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1da851b3b56796b5df83e773031eebfc05d90d7f983d0b1e5b7bb3e6ae06a4b8" Oct 30 13:28:31.365094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:31.364911 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2" Oct 30 13:28:31.743890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:31.743809 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:31.897610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:31.897496 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:32.046422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:32.043265 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6"] Oct 30 13:28:32.046422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:32.043543 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" containerID="cri-o://eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411" gracePeriod=30 Oct 30 13:28:32.120579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:32.120541 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:32.788383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:32.788313 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:32.975676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:32.975642 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:33.197061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:33.196897 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:33.375616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:33.375554 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:33.547627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:33.547596 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:33.785876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:33.785840 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:34.634780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:34.634742 5049 patch_prober.go:29] interesting pod/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:28:34.635170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:34.634816 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:28:34.851987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:34.851946 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:36.690913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:36.689819 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:37.362892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.362865 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:28:37.396211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396183 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-certs\") pod \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " Oct 30 13:28:37.396402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396229 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istiod-ca-cert\") pod \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " Oct 30 13:28:37.396402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396255 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-envoy\") pod \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " Oct 30 13:28:37.396402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396297 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-kube-api-access-bxx6t\" (UniqueName: \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-kube-api-access-bxx6t\") pod \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " Oct 30 13:28:37.396402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396361 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-token\" (UniqueName: \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-token\") pod \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " Oct 30 13:28:37.396640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396420 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-credential-socket\") pod \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " Oct 30 13:28:37.396640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396469 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-podinfo\") pod \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " Oct 30 13:28:37.396640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396510 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-certs" pod "38e9360b-c35c-47bd-b0fe-1ccad4274ab9" (UID: "38e9360b-c35c-47bd-b0fe-1ccad4274ab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:37.396640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396597 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-socket\") pod \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " Oct 30 13:28:37.396854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396669 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-data\") pod \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\" (UID: \"38e9360b-c35c-47bd-b0fe-1ccad4274ab9\") " Oct 30 13:28:37.396854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396755 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istiod-ca-cert" pod "38e9360b-c35c-47bd-b0fe-1ccad4274ab9" (UID: "38e9360b-c35c-47bd-b0fe-1ccad4274ab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:37.396854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396765 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-credential-socket" pod "38e9360b-c35c-47bd-b0fe-1ccad4274ab9" (UID: "38e9360b-c35c-47bd-b0fe-1ccad4274ab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:37.397024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.396963 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-data" pod "38e9360b-c35c-47bd-b0fe-1ccad4274ab9" (UID: "38e9360b-c35c-47bd-b0fe-1ccad4274ab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:37.397100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.397027 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-socket" pod "38e9360b-c35c-47bd-b0fe-1ccad4274ab9" (UID: "38e9360b-c35c-47bd-b0fe-1ccad4274ab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:37.397604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.397580 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.397725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.397608 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.397725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.397623 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.397725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.397636 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.397725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.397648 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.398642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.398614 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-envoy" pod "38e9360b-c35c-47bd-b0fe-1ccad4274ab9" (UID: "38e9360b-c35c-47bd-b0fe-1ccad4274ab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:37.398775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.398754 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-podinfo" pod "38e9360b-c35c-47bd-b0fe-1ccad4274ab9" (UID: "38e9360b-c35c-47bd-b0fe-1ccad4274ab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:28:37.398849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.398795 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-kube-api-access-bxx6t" pod "38e9360b-c35c-47bd-b0fe-1ccad4274ab9" (UID: "38e9360b-c35c-47bd-b0fe-1ccad4274ab9"). InnerVolumeSpecName "kube-api-access-bxx6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:37.398965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.398947 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-token" pod "38e9360b-c35c-47bd-b0fe-1ccad4274ab9" (UID: "38e9360b-c35c-47bd-b0fe-1ccad4274ab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:37.435633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.435593 5049 generic.go:356] "Generic (PLEG): container finished" podID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerID="eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411" exitCode=0 Oct 30 13:28:37.435839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.435815 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" event={"ID":"38e9360b-c35c-47bd-b0fe-1ccad4274ab9","Type":"ContainerDied","Data":"eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411"} Oct 30 13:28:37.436059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.436032 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" event={"ID":"38e9360b-c35c-47bd-b0fe-1ccad4274ab9","Type":"ContainerDied","Data":"7ec5e48ce3404c16bf87580932fb675d26e3ee65ca57a22fa894b7c96fdf7256"} Oct 30 13:28:37.436171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.436078 5049 scope.go:122] "RemoveContainer" containerID="eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411" Oct 30 13:28:37.436268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.436250 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6" Oct 30 13:28:37.460016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.459996 5049 scope.go:122] "RemoveContainer" containerID="29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c" Oct 30 13:28:37.477633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.477606 5049 scope.go:122] "RemoveContainer" containerID="eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411" Oct 30 13:28:37.477986 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:28:37.477957 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411\": container with ID starting with eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411 not found: ID does not exist" containerID="eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411" Oct 30 13:28:37.478069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.478001 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411"} err="failed to get container status \"eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411\": rpc error: code = NotFound desc = could not find container \"eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411\": container with ID starting with eaced63dc6afa0647ca94c982991ba88e2d9a9be956f6bea16535f242b391411 not found: ID does not exist" Oct 30 13:28:37.478069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.478024 5049 scope.go:122] "RemoveContainer" containerID="29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c" Oct 30 13:28:37.478359 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:28:37.478335 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c\": container with ID starting with 29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c not found: ID does not exist" containerID="29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c" Oct 30 13:28:37.478433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.478369 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c"} err="failed to get container status \"29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c\": rpc error: code = NotFound desc = could not find container \"29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c\": container with ID starting with 29779c58d734dddc4ed7d6a3d08afaaee998aa44da15ce98c618285e117d153c not found: ID does not exist" Oct 30 13:28:37.495088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.495066 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6"] Oct 30 13:28:37.499219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.499185 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.499219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.499217 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxx6t\" (UniqueName: \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-kube-api-access-bxx6t\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.499421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.499234 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.499421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.499248 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e9360b-c35c-47bd-b0fe-1ccad4274ab9-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.499649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.499626 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-09e4c0-openshift-default-6d59c4fbcf-lsrr6"] Oct 30 13:28:37.903594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:37.903555 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:38.116512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:38.116459 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:38.365971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:38.364184 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:38.800847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:38.800816 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:39.342514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:39.342481 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" path="/var/lib/kubelet/pods/38e9360b-c35c-47bd-b0fe-1ccad4274ab9/volumes" Oct 30 13:28:39.431972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:39.431937 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:40.036043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.036004 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:40.089024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.088981 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h"] Oct 30 13:28:40.089886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.089859 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3f1089e5-8890-4d38-b17e-78b2b651e392" containerName="revoke-keys" Oct 30 13:28:40.089886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.089880 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1089e5-8890-4d38-b17e-78b2b651e392" containerName="revoke-keys" Oct 30 13:28:40.090106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.089895 5049 cpu_manager.go:414] "containerMap: removing container" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" Oct 30 13:28:40.090106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.089901 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" Oct 30 13:28:40.090106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.090024 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" Oct 30 13:28:40.090106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.090044 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1089e5-8890-4d38-b17e-78b2b651e392" containerName="revoke-keys" Oct 30 13:28:40.095789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.095764 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:28:40.097779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.097753 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb"] Oct 30 13:28:40.098519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.098482 5049 cpu_manager.go:414] "containerMap: removing container" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" Oct 30 13:28:40.098519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.098510 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" Oct 30 13:28:40.098717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.098702 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e9360b-c35c-47bd-b0fe-1ccad4274ab9" containerName="istio-proxy" Oct 30 13:28:40.102678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.102657 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:28:40.115839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.115818 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-9cbcbc\"" Oct 30 13:28:40.122240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.122219 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h"] Oct 30 13:28:40.126466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.126444 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-9cbcbc-dockercfg-p4d8m\"" Oct 30 13:28:40.129198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.129176 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb"] Oct 30 13:28:40.153309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.153269 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h\" (UID: \"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:28:40.153525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.153354 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzcqg\" (UniqueName: \"kubernetes.io/projected/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-kube-api-access-lzcqg\") pod \"payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h\" (UID: \"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:28:40.153525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.153387 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbc92\" (UniqueName: \"kubernetes.io/projected/22699b7b-4d79-427b-a469-a8d2d87a7e51-kube-api-access-cbc92\") pod \"payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb\" (UID: \"22699b7b-4d79-427b-a469-a8d2d87a7e51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:28:40.153663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.153544 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22699b7b-4d79-427b-a469-a8d2d87a7e51-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb\" (UID: \"22699b7b-4d79-427b-a469-a8d2d87a7e51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:28:40.247347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.242314 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:40.321346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.320524 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h\" (UID: \"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:28:40.321346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.320691 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h\" (UID: \"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:28:40.321346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.320785 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzcqg\" (UniqueName: \"kubernetes.io/projected/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-kube-api-access-lzcqg\") pod \"payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h\" (UID: \"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:28:40.321346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.320836 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbc92\" (UniqueName: \"kubernetes.io/projected/22699b7b-4d79-427b-a469-a8d2d87a7e51-kube-api-access-cbc92\") pod \"payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb\" (UID: \"22699b7b-4d79-427b-a469-a8d2d87a7e51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:28:40.321346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.321147 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22699b7b-4d79-427b-a469-a8d2d87a7e51-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb\" (UID: \"22699b7b-4d79-427b-a469-a8d2d87a7e51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:28:40.340347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.336655 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22699b7b-4d79-427b-a469-a8d2d87a7e51-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb\" (UID: \"22699b7b-4d79-427b-a469-a8d2d87a7e51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:28:40.363346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.346490 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbc92\" (UniqueName: \"kubernetes.io/projected/22699b7b-4d79-427b-a469-a8d2d87a7e51-kube-api-access-cbc92\") pod \"payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb\" (UID: \"22699b7b-4d79-427b-a469-a8d2d87a7e51\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:28:40.364922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.364891 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzcqg\" (UniqueName: \"kubernetes.io/projected/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-kube-api-access-lzcqg\") pod \"payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h\" (UID: \"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:28:40.418343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.414463 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:28:40.430146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.430114 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:28:40.803345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.796431 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h"] Oct 30 13:28:40.833535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:40.833502 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb"] Oct 30 13:28:40.834666 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:28:40.834633 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22699b7b_4d79_427b_a469_a8d2d87a7e51.slice/crio-ed39d7efa8e695881b3bfd49418609ef003a81f2a19a997d832df69d7fcc3923 WatchSource:0}: Error finding container ed39d7efa8e695881b3bfd49418609ef003a81f2a19a997d832df69d7fcc3923: Status 404 returned error can't find the container with id ed39d7efa8e695881b3bfd49418609ef003a81f2a19a997d832df69d7fcc3923 Oct 30 13:28:41.525796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:41.525690 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" event={"ID":"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e","Type":"ContainerStarted","Data":"874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08"} Oct 30 13:28:41.525796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:41.525757 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" event={"ID":"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e","Type":"ContainerStarted","Data":"df59ca7d0631a41d5f68f72bcd66ae5f6c9a990c1857aa2c6fd3e79b6764548c"} Oct 30 13:28:41.527341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:41.527305 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:28:41.528103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:41.528074 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" event={"ID":"22699b7b-4d79-427b-a469-a8d2d87a7e51","Type":"ContainerStarted","Data":"66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c"} Oct 30 13:28:41.528267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:41.528124 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" event={"ID":"22699b7b-4d79-427b-a469-a8d2d87a7e51","Type":"ContainerStarted","Data":"ed39d7efa8e695881b3bfd49418609ef003a81f2a19a997d832df69d7fcc3923"} Oct 30 13:28:41.528425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:41.528339 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:28:41.572497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:41.572429 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" podStartSLOduration=1.572410003 podStartE2EDuration="1.572410003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:41.571721909 +0000 UTC m=+2228.386916329" watchObservedRunningTime="2026-10-30 13:28:41.572410003 +0000 UTC m=+2228.387604428" Oct 30 13:28:41.872590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:41.871769 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:42.437072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:42.430957 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" podUID="5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" containerName="main" containerID="cri-o://4eec222047812262ef9441951c85b1782d491b8d034aa394fe758082d81e6536" gracePeriod=45 Oct 30 13:28:42.511482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:42.507143 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" podUID="5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.210:8000/ready\": dial tcp 10.132.0.210:8000: connect: connection refused" Oct 30 13:28:43.558340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.558098 5049 generic.go:356] "Generic (PLEG): container finished" podID="5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" containerID="4eec222047812262ef9441951c85b1782d491b8d034aa394fe758082d81e6536" exitCode=0 Oct 30 13:28:43.558340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.558163 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" event={"ID":"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1","Type":"ContainerDied","Data":"4eec222047812262ef9441951c85b1782d491b8d034aa394fe758082d81e6536"} Oct 30 13:28:43.651200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.651171 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:28:43.693226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.693170 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" podStartSLOduration=3.693155101 podStartE2EDuration="3.693155101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:41.613760993 +0000 UTC m=+2228.428955417" watchObservedRunningTime="2026-10-30 13:28:43.693155101 +0000 UTC m=+2230.508349524" Oct 30 13:28:43.736401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.735456 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-home\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-home\") pod \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " Oct 30 13:28:43.736401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.735517 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tmp-dir\") pod \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " Oct 30 13:28:43.736401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.735643 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-model-cache\") pod \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " Oct 30 13:28:43.736401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.735712 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tls-certs\") pod \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " Oct 30 13:28:43.736401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.735827 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-dshm\") pod \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " Oct 30 13:28:43.736401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.735884 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-kube-api-access-rqm7w\" (UniqueName: \"kubernetes.io/projected/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-kube-api-access-rqm7w\") pod \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\" (UID: \"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1\") " Oct 30 13:28:43.737402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.736958 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-model-cache" pod "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" (UID: "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:43.737402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.737156 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-home" pod "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" (UID: "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:43.737585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.737401 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tmp-dir" pod "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" (UID: "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:43.740674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.740643 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-dshm" pod "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" (UID: "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:43.740782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.740635 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tls-certs" pod "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" (UID: "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:43.746439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.742069 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-kube-api-access-rqm7w" pod "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" (UID: "5bf0a3d8-f0b5-4855-a9b9-a552f61415b1"). InnerVolumeSpecName "kube-api-access-rqm7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:43.842346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.841628 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:43.842346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.841663 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqm7w\" (UniqueName: \"kubernetes.io/projected/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-kube-api-access-rqm7w\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:43.842346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.841675 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:43.842346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.841683 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:43.842346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.841693 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:43.842346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:43.841701 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:28:44.039421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:44.032276 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:44.497154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:44.497113 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" podUID="5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.210:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:28:44.497364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:44.497214 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:28:44.569039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:44.569008 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" Oct 30 13:28:44.569515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:44.569045 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr" event={"ID":"5bf0a3d8-f0b5-4855-a9b9-a552f61415b1","Type":"ContainerDied","Data":"ade23581de8a58654d1fb8e895c6f0bfbd0944a08b598502467cc66497a7a5b8"} Oct 30 13:28:44.569515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:44.569100 5049 scope.go:122] "RemoveContainer" containerID="4eec222047812262ef9441951c85b1782d491b8d034aa394fe758082d81e6536" Oct 30 13:28:44.636427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:44.636390 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr"] Oct 30 13:28:44.640814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:44.640792 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-376c89/test-model-376c89-kserve-7cdc64f4cf-nqgrr"] Oct 30 13:28:45.345507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:45.345478 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" path="/var/lib/kubelet/pods/5bf0a3d8-f0b5-4855-a9b9-a552f61415b1/volumes" Oct 30 13:28:45.765306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:45.765216 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:47.751356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:47.750996 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:49.576395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:49.576231 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:51.559842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:51.559805 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:52.549475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:52.549447 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:28:52.549689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:52.549528 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:28:52.657286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:52.657249 5049 generic.go:356] "Generic (PLEG): container finished" podID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerID="8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b" exitCode=0 Oct 30 13:28:52.657807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:52.657376 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" event={"ID":"27db492b-5ede-4ac7-99e5-06c5bc19051f","Type":"ContainerDied","Data":"8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b"} Oct 30 13:28:52.657928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:52.657904 5049 scope.go:122] "RemoveContainer" containerID="8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b" Oct 30 13:28:52.660245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:52.660213 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:52.660411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:52.660303 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:52.660411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:52.660372 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:53.033302 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:53.033214 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9"] Oct 30 13:28:53.040102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:53.040064 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee526181-gxxh9"] Oct 30 13:28:53.311347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:53.311254 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:53.351984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:53.351940 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abc99ff-a7fb-4801-b89b-e4ba0db7b480" path="/var/lib/kubelet/pods/6abc99ff-a7fb-4801-b89b-e4ba0db7b480/volumes" Oct 30 13:28:53.668061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:53.667972 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" event={"ID":"27db492b-5ede-4ac7-99e5-06c5bc19051f","Type":"ContainerStarted","Data":"05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0"} Oct 30 13:28:54.057861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:54.057829 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:28:54.058155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:54.058130 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 13:28:54.058215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:54.058186 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 13:28:55.059254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:55.059212 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 13:28:55.059843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:55.059289 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 13:28:55.115347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:55.110095 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:56.058689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:56.058657 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 13:28:56.058876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:56.058708 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 13:28:56.774346 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:28:56.773730 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:28:57.058739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:57.058707 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 13:28:57.058954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:57.058770 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 13:28:57.209372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:57.205747 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:57.892757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:57.892706 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:58.058262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:58.058226 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 13:28:58.058488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:58.058284 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 13:28:58.348427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:58.348391 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:58.844346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:58.839368 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:59.058257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:59.058225 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 13:28:59.058699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:59.058274 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 13:28:59.154695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:59.154607 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:59.306948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:59.306913 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:59.506198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:28:59.506106 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:29:00.063503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:00.063473 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:29:00.064440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:00.064417 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:29:01.231251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:01.231217 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:02.243579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.243531 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt"] Oct 30 13:29:02.244686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.244662 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" containerName="main" Oct 30 13:29:02.244812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.244687 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" containerName="main" Oct 30 13:29:02.244965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.244951 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf0a3d8-f0b5-4855-a9b9-a552f61415b1" containerName="main" Oct 30 13:29:02.249697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.249675 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.280524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.280489 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt"] Oct 30 13:29:02.435125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.435088 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.435355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.435142 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.435426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.435359 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe8ad4c-51e3-416f-a972-dfa870688e69-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.435480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.435443 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.435480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.435471 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q79tw\" (UniqueName: \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-kube-api-access-q79tw\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.435645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.435624 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.436091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.435872 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.436091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.435906 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.436311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.436167 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.537484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.537383 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.537484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.537461 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.537484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.537493 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.537804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.537549 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe8ad4c-51e3-416f-a972-dfa870688e69-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.537804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.537587 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.537804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.537609 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q79tw\" (UniqueName: \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-kube-api-access-q79tw\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.537804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.537655 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.538023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.537823 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.538023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.537851 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.538023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.537931 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.538275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.538250 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.538385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.538304 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.538468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.538398 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe8ad4c-51e3-416f-a972-dfa870688e69-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.538608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.538584 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.541449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.541429 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.541677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.541655 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.561138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.561109 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.561792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.561767 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q79tw\" (UniqueName: \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-kube-api-access-q79tw\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.564988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.564969 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:02.786054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.786022 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk"] Oct 30 13:29:02.786247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.786214 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" podUID="08571487-c812-4d29-9dca-7e20a422a2ac" containerName="maas-api" containerID="cri-o://fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19" gracePeriod=30 Oct 30 13:29:02.903313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.903286 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt"] Oct 30 13:29:02.903865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.903815 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:02.904496 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:29:02.904472 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe8ad4c_51e3_416f_a972_dfa870688e69.slice/crio-8d6a85442eeee22bcbf11f4d4f6021357e6284d947ad10cd8ca4219b8a60bca9 WatchSource:0}: Error finding container 8d6a85442eeee22bcbf11f4d4f6021357e6284d947ad10cd8ca4219b8a60bca9: Status 404 returned error can't find the container with id 8d6a85442eeee22bcbf11f4d4f6021357e6284d947ad10cd8ca4219b8a60bca9 Oct 30 13:29:02.907638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.907601 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:02.907757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.907687 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:02.907757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:02.907732 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:03.151344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.137614 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:29:03.253584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.253545 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-maas-api-tls\") pod \"08571487-c812-4d29-9dca-7e20a422a2ac\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " Oct 30 13:29:03.254132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.253622 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08571487-c812-4d29-9dca-7e20a422a2ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08571487-c812-4d29-9dca-7e20a422a2ac-openshift-service-ca\") pod \"08571487-c812-4d29-9dca-7e20a422a2ac\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " Oct 30 13:29:03.254132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.253834 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-metrics-certs\") pod \"08571487-c812-4d29-9dca-7e20a422a2ac\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " Oct 30 13:29:03.254132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.254002 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08571487-c812-4d29-9dca-7e20a422a2ac-openshift-service-ca" pod "08571487-c812-4d29-9dca-7e20a422a2ac" (UID: "08571487-c812-4d29-9dca-7e20a422a2ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:03.254381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.254186 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08571487-c812-4d29-9dca-7e20a422a2ac-kube-api-access-ks8vn\" (UniqueName: \"kubernetes.io/projected/08571487-c812-4d29-9dca-7e20a422a2ac-kube-api-access-ks8vn\") pod \"08571487-c812-4d29-9dca-7e20a422a2ac\" (UID: \"08571487-c812-4d29-9dca-7e20a422a2ac\") " Oct 30 13:29:03.255024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.254998 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08571487-c812-4d29-9dca-7e20a422a2ac-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:03.256067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.256040 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-maas-api-tls" pod "08571487-c812-4d29-9dca-7e20a422a2ac" (UID: "08571487-c812-4d29-9dca-7e20a422a2ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:03.256217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.256199 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08571487-c812-4d29-9dca-7e20a422a2ac-kube-api-access-ks8vn" pod "08571487-c812-4d29-9dca-7e20a422a2ac" (UID: "08571487-c812-4d29-9dca-7e20a422a2ac"). InnerVolumeSpecName "kube-api-access-ks8vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:03.256718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.256697 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-metrics-certs" pod "08571487-c812-4d29-9dca-7e20a422a2ac" (UID: "08571487-c812-4d29-9dca-7e20a422a2ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:03.356373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.356296 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:03.356373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.356347 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08571487-c812-4d29-9dca-7e20a422a2ac-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:03.356373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.356362 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8vn\" (UniqueName: \"kubernetes.io/projected/08571487-c812-4d29-9dca-7e20a422a2ac-kube-api-access-ks8vn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:03.398285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.398159 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:03.786451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.786349 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:03.843239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.843194 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" event={"ID":"4fe8ad4c-51e3-416f-a972-dfa870688e69","Type":"ContainerStarted","Data":"8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5"} Oct 30 13:29:03.843462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.843293 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" event={"ID":"4fe8ad4c-51e3-416f-a972-dfa870688e69","Type":"ContainerStarted","Data":"8d6a85442eeee22bcbf11f4d4f6021357e6284d947ad10cd8ca4219b8a60bca9"} Oct 30 13:29:03.844796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.844770 5049 generic.go:356] "Generic (PLEG): container finished" podID="08571487-c812-4d29-9dca-7e20a422a2ac" containerID="fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19" exitCode=0 Oct 30 13:29:03.844952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.844836 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" Oct 30 13:29:03.844952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.844849 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" event={"ID":"08571487-c812-4d29-9dca-7e20a422a2ac","Type":"ContainerDied","Data":"fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19"} Oct 30 13:29:03.844952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.844897 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk" event={"ID":"08571487-c812-4d29-9dca-7e20a422a2ac","Type":"ContainerDied","Data":"2e2e63b18538c3528e14c1ab956d120265595cdef2be6d145110929a4a4eff82"} Oct 30 13:29:03.844952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.844919 5049 scope.go:122] "RemoveContainer" containerID="fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19" Oct 30 13:29:03.856565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.856542 5049 scope.go:122] "RemoveContainer" containerID="fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19" Oct 30 13:29:03.856856 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:29:03.856836 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19\": container with ID starting with fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19 not found: ID does not exist" containerID="fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19" Oct 30 13:29:03.856927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.856866 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19"} err="failed to get container status \"fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19\": rpc error: code = NotFound desc = could not find container \"fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19\": container with ID starting with fa7c06fae76a586a7fc2755c3a178c925469f5857a400e6841165de8974c9e19 not found: ID does not exist" Oct 30 13:29:03.897643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.897592 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" podStartSLOduration=1.897577664 podStartE2EDuration="1.897577664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:03.894294994 +0000 UTC m=+2250.709489419" watchObservedRunningTime="2026-10-30 13:29:03.897577664 +0000 UTC m=+2250.712772152" Oct 30 13:29:03.946409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.946374 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b"] Oct 30 13:29:03.957924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.957889 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-47628d-r7v9b"] Oct 30 13:29:03.961929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.961903 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk"] Oct 30 13:29:03.965955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:03.965929 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47628d-587b5897c6-d6qpk"] Oct 30 13:29:04.031279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.031248 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:04.085640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.085610 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8"] Oct 30 13:29:04.086431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.086408 5049 cpu_manager.go:414] "containerMap: removing container" podUID="08571487-c812-4d29-9dca-7e20a422a2ac" containerName="maas-api" Oct 30 13:29:04.086431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.086428 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08571487-c812-4d29-9dca-7e20a422a2ac" containerName="maas-api" Oct 30 13:29:04.086648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.086596 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="08571487-c812-4d29-9dca-7e20a422a2ac" containerName="maas-api" Oct 30 13:29:04.092634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.092297 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.152006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.151389 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8"] Oct 30 13:29:04.188464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.188430 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.188631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.188478 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.188631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.188607 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qwqh\" (UniqueName: \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-kube-api-access-4qwqh\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.188720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.188640 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.188807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.188789 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.188845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.188837 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.188906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.188893 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.188953 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.188942 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.189002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.188987 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.290411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290376 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.290411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290419 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.290967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290446 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.290967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290475 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.290967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290534 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qwqh\" (UniqueName: \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-kube-api-access-4qwqh\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.290967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290555 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.290967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290705 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.290967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290775 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.290967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290800 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.290967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290808 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.291468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.290988 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.291468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.291068 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.291468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.291210 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.291468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.291285 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.292879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.292858 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.293194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.293174 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.338924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.338862 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.340053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.340024 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qwqh\" (UniqueName: \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-kube-api-access-4qwqh\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.448478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.448446 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:04.454644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.454620 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:04.568743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.567623 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:04.569029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.569009 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 13:29:04.569128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.569061 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 13:29:04.839086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.839055 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8"] Oct 30 13:29:04.839434 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:29:04.839405 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e60a821_b3b0_4653_b7e9_8ab6cb2818a7.slice/crio-8b581ae5a91d5492e6cbceb6389a501bc6ffdf6e6c2bd0317d388ec62c01323f WatchSource:0}: Error finding container 8b581ae5a91d5492e6cbceb6389a501bc6ffdf6e6c2bd0317d388ec62c01323f: Status 404 returned error can't find the container with id 8b581ae5a91d5492e6cbceb6389a501bc6ffdf6e6c2bd0317d388ec62c01323f Oct 30 13:29:04.843306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.843272 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:04.843400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.843373 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:04.843441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.843415 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:04.856801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:04.856770 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" event={"ID":"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7","Type":"ContainerStarted","Data":"8b581ae5a91d5492e6cbceb6389a501bc6ffdf6e6c2bd0317d388ec62c01323f"} Oct 30 13:29:05.096054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.094440 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:05.268402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.266469 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:05.581031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.580994 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 13:29:05.581647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.581612 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 13:29:05.592849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.592818 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08571487-c812-4d29-9dca-7e20a422a2ac" path="/var/lib/kubelet/pods/08571487-c812-4d29-9dca-7e20a422a2ac/volumes" Oct 30 13:29:05.603555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.603524 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52653f4c-bcb3-4095-9cda-0ff663771097" path="/var/lib/kubelet/pods/52653f4c-bcb3-4095-9cda-0ff663771097/volumes" Oct 30 13:29:05.639345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.638536 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp"] Oct 30 13:29:05.657750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.657721 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp"] Oct 30 13:29:05.658012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.657991 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:29:05.673024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.672667 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4b0ea9\"" Oct 30 13:29:05.681570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.681544 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4b0ea9-dockercfg-r9ddf\"" Oct 30 13:29:05.777972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.777940 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvsms\" (UniqueName: \"kubernetes.io/projected/e0fd7d58-f16f-47d9-8265-0cb67014efa1-kube-api-access-cvsms\") pod \"payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp\" (UID: \"e0fd7d58-f16f-47d9-8265-0cb67014efa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:29:05.778316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.778299 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fd7d58-f16f-47d9-8265-0cb67014efa1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp\" (UID: \"e0fd7d58-f16f-47d9-8265-0cb67014efa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:29:05.871789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.871706 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" event={"ID":"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7","Type":"ContainerStarted","Data":"626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0"} Oct 30 13:29:05.879639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.879610 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvsms\" (UniqueName: \"kubernetes.io/projected/e0fd7d58-f16f-47d9-8265-0cb67014efa1-kube-api-access-cvsms\") pod \"payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp\" (UID: \"e0fd7d58-f16f-47d9-8265-0cb67014efa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:29:05.879779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.879758 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fd7d58-f16f-47d9-8265-0cb67014efa1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp\" (UID: \"e0fd7d58-f16f-47d9-8265-0cb67014efa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:29:05.880614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.880597 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fd7d58-f16f-47d9-8265-0cb67014efa1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp\" (UID: \"e0fd7d58-f16f-47d9-8265-0cb67014efa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:29:05.903352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.902958 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvsms\" (UniqueName: \"kubernetes.io/projected/e0fd7d58-f16f-47d9-8265-0cb67014efa1-kube-api-access-cvsms\") pod \"payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp\" (UID: \"e0fd7d58-f16f-47d9-8265-0cb67014efa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:29:05.920351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.916811 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" podStartSLOduration=1.916793215 podStartE2EDuration="1.916793215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:05.915794691 +0000 UTC m=+2252.730989127" watchObservedRunningTime="2026-10-30 13:29:05.916793215 +0000 UTC m=+2252.731987640" Oct 30 13:29:05.977151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:05.977123 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:29:06.279346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.279203 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:06.473382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.470394 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp"] Oct 30 13:29:06.473382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.470491 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:06.473382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.472955 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 13:29:06.473382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.473002 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 13:29:06.569470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.568719 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 13:29:06.569470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.568781 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 13:29:06.896574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.896533 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" event={"ID":"e0fd7d58-f16f-47d9-8265-0cb67014efa1","Type":"ContainerStarted","Data":"3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d"} Oct 30 13:29:06.896974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.896617 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" event={"ID":"e0fd7d58-f16f-47d9-8265-0cb67014efa1","Type":"ContainerStarted","Data":"6689aadf7e931bf5d14601dcb33f9d07515b30a4b618fd6ac71029f8490b7137"} Oct 30 13:29:06.896974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.896742 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:29:06.971823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:06.971758 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" podStartSLOduration=1.971738805 podStartE2EDuration="1.971738805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:06.969781316 +0000 UTC m=+2253.784975740" watchObservedRunningTime="2026-10-30 13:29:06.971738805 +0000 UTC m=+2253.786933229" Oct 30 13:29:07.069633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:07.061705 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l"] Oct 30 13:29:07.069633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:07.068113 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240e9cfd-tcw8l"] Oct 30 13:29:07.372658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:07.372081 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="958d0bfa-33b3-446b-84d1-9d8d34eb72b1" path="/var/lib/kubelet/pods/958d0bfa-33b3-446b-84d1-9d8d34eb72b1/volumes" Oct 30 13:29:07.456344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:07.455731 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 13:29:07.456344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:07.455789 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 13:29:07.568345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:07.567779 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 13:29:07.568345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:07.567844 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 13:29:08.295372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:08.294303 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:08.463772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:08.463734 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 13:29:08.464057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:08.464026 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 13:29:08.465006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:08.464969 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:08.575415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:08.573042 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 13:29:08.575415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:08.573101 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 13:29:09.455899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:09.455854 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 13:29:09.456413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:09.455915 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 13:29:09.566345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:09.566302 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 13:29:09.566528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:09.566364 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 13:29:10.455527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:10.455490 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 13:29:10.455695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:10.455544 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 13:29:10.528346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:10.525853 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:10.634345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:10.627634 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:10.634345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:10.628441 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:10.634345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:10.631531 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:29:11.314314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:11.314263 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:11.455546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:11.455512 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 13:29:11.455736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:11.455573 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 13:29:11.472041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:11.471998 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:11.752709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:11.752602 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q"] Oct 30 13:29:11.753239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:11.752847 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" containerID="cri-o://edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1" gracePeriod=30 Oct 30 13:29:11.872315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:11.871969 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:12.166577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.166544 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb"] Oct 30 13:29:12.174033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.174005 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.188630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.188599 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:29:12.188827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.188785 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:29:12.209198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.206724 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb"] Oct 30 13:29:12.276354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.276307 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:12.280820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.280789 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.280940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.280907 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.281007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.280984 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k294\" (UniqueName: \"kubernetes.io/projected/e11fca44-948f-4709-9632-0646ab90f3d5-kube-api-access-6k294\") pod \"maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.382082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.382043 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k294\" (UniqueName: \"kubernetes.io/projected/e11fca44-948f-4709-9632-0646ab90f3d5-kube-api-access-6k294\") pod \"maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.382428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.382405 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.382608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.382586 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.383195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.383171 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.385288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.385266 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.406372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.406344 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k294\" (UniqueName: \"kubernetes.io/projected/e11fca44-948f-4709-9632-0646ab90f3d5-kube-api-access-6k294\") pod \"maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.460253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.460175 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:12.460711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.460683 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:12.461568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.461550 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:29:12.528197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.528168 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:12.948197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:12.948171 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb"] Oct 30 13:29:12.950622 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:29:12.950597 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11fca44_948f_4709_9632_0646ab90f3d5.slice/crio-14f5fedcd17cb518c266dba8ba44960d79acb21f480bdfa5cef6b5aa73d5f33e WatchSource:0}: Error finding container 14f5fedcd17cb518c266dba8ba44960d79acb21f480bdfa5cef6b5aa73d5f33e: Status 404 returned error can't find the container with id 14f5fedcd17cb518c266dba8ba44960d79acb21f480bdfa5cef6b5aa73d5f33e Oct 30 13:29:13.016472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:13.016434 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" event={"ID":"e11fca44-948f-4709-9632-0646ab90f3d5","Type":"ContainerStarted","Data":"14f5fedcd17cb518c266dba8ba44960d79acb21f480bdfa5cef6b5aa73d5f33e"} Oct 30 13:29:13.198998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:13.198888 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:13.290500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:13.290469 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2"] Oct 30 13:29:13.298728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:13.298521 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4651e8-9j4n2"] Oct 30 13:29:13.337498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:13.337467 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1089e5-8890-4d38-b17e-78b2b651e392" path="/var/lib/kubelet/pods/3f1089e5-8890-4d38-b17e-78b2b651e392/volumes" Oct 30 13:29:13.382408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:13.382361 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:14.029082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:14.029046 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" event={"ID":"e11fca44-948f-4709-9632-0646ab90f3d5","Type":"ContainerStarted","Data":"d352c6b78996abe7227f2056d70157df4ac7d03a6d6f0fe42f3752f3831596d1"} Oct 30 13:29:14.078827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:14.078797 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:14.085676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:14.085629 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" podStartSLOduration=2.085616684 podStartE2EDuration="2.085616684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:14.0822784 +0000 UTC m=+2260.897472823" watchObservedRunningTime="2026-10-30 13:29:14.085616684 +0000 UTC m=+2260.900811107" Oct 30 13:29:15.039217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:15.039182 5049 generic.go:356] "Generic (PLEG): container finished" podID="e11fca44-948f-4709-9632-0646ab90f3d5" containerID="d352c6b78996abe7227f2056d70157df4ac7d03a6d6f0fe42f3752f3831596d1" exitCode=0 Oct 30 13:29:15.039668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:15.039274 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" event={"ID":"e11fca44-948f-4709-9632-0646ab90f3d5","Type":"ContainerDied","Data":"d352c6b78996abe7227f2056d70157df4ac7d03a6d6f0fe42f3752f3831596d1"} Oct 30 13:29:15.497873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:15.497835 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:16.228762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:16.228736 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:16.285799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:16.285753 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-openshift-ca\") pod \"e11fca44-948f-4709-9632-0646ab90f3d5\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " Oct 30 13:29:16.286015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:16.285856 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-xks-ca\") pod \"e11fca44-948f-4709-9632-0646ab90f3d5\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " Oct 30 13:29:16.286015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:16.285992 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11fca44-948f-4709-9632-0646ab90f3d5-kube-api-access-6k294\" (UniqueName: \"kubernetes.io/projected/e11fca44-948f-4709-9632-0646ab90f3d5-kube-api-access-6k294\") pod \"e11fca44-948f-4709-9632-0646ab90f3d5\" (UID: \"e11fca44-948f-4709-9632-0646ab90f3d5\") " Oct 30 13:29:16.287675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:16.287647 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-openshift-ca" pod "e11fca44-948f-4709-9632-0646ab90f3d5" (UID: "e11fca44-948f-4709-9632-0646ab90f3d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:16.289803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:16.289777 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-xks-ca" pod "e11fca44-948f-4709-9632-0646ab90f3d5" (UID: "e11fca44-948f-4709-9632-0646ab90f3d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:16.289803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:16.289794 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11fca44-948f-4709-9632-0646ab90f3d5-kube-api-access-6k294" pod "e11fca44-948f-4709-9632-0646ab90f3d5" (UID: "e11fca44-948f-4709-9632-0646ab90f3d5"). InnerVolumeSpecName "kube-api-access-6k294". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:16.388738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:16.388707 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:16.388738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:16.388735 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11fca44-948f-4709-9632-0646ab90f3d5-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:16.389000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:16.388749 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k294\" (UniqueName: \"kubernetes.io/projected/e11fca44-948f-4709-9632-0646ab90f3d5-kube-api-access-6k294\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.036803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.036774 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:29:17.059257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.059222 5049 generic.go:356] "Generic (PLEG): container finished" podID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerID="edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1" exitCode=0 Oct 30 13:29:17.059455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.059272 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" Oct 30 13:29:17.059455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.059303 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" event={"ID":"1a94e7e5-3c0e-4f97-a16c-622e809fcd71","Type":"ContainerDied","Data":"edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1"} Oct 30 13:29:17.059455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.059395 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q" event={"ID":"1a94e7e5-3c0e-4f97-a16c-622e809fcd71","Type":"ContainerDied","Data":"06c7916bf1d50ebcc22cbdf1999e344cd71bc4bb08eb21c9c25a14b9e7c83818"} Oct 30 13:29:17.059455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.059425 5049 scope.go:122] "RemoveContainer" containerID="edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1" Oct 30 13:29:17.061146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.061128 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" Oct 30 13:29:17.061293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.061168 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb" event={"ID":"e11fca44-948f-4709-9632-0646ab90f3d5","Type":"ContainerDied","Data":"14f5fedcd17cb518c266dba8ba44960d79acb21f480bdfa5cef6b5aa73d5f33e"} Oct 30 13:29:17.061293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.061214 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f5fedcd17cb518c266dba8ba44960d79acb21f480bdfa5cef6b5aa73d5f33e" Oct 30 13:29:17.072268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.072248 5049 scope.go:122] "RemoveContainer" containerID="795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c" Oct 30 13:29:17.083751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.083725 5049 scope.go:122] "RemoveContainer" containerID="edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1" Oct 30 13:29:17.084020 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:29:17.084000 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1\": container with ID starting with edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1 not found: ID does not exist" containerID="edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1" Oct 30 13:29:17.084108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.084031 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1"} err="failed to get container status \"edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1\": rpc error: code = NotFound desc = could not find container \"edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1\": container with ID starting with edaf36b306e9b076e1e5d942b5d9c83e877557945e1619779db38146a820d6e1 not found: ID does not exist" Oct 30 13:29:17.084108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.084060 5049 scope.go:122] "RemoveContainer" containerID="795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c" Oct 30 13:29:17.084359 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:29:17.084331 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c\": container with ID starting with 795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c not found: ID does not exist" containerID="795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c" Oct 30 13:29:17.084406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.084370 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c"} err="failed to get container status \"795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c\": rpc error: code = NotFound desc = could not find container \"795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c\": container with ID starting with 795240a94e3e5d3bac3157eadddf0a9ca71853a5ce95d2905d94577510d5df9c not found: ID does not exist" Oct 30 13:29:17.099941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.099911 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-credential-socket\") pod \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " Oct 30 13:29:17.101103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.100035 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-podinfo\") pod \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " Oct 30 13:29:17.101103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.100136 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-envoy\") pod \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " Oct 30 13:29:17.101103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.100172 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-certs\") pod \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " Oct 30 13:29:17.101103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.100263 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-data\") pod \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " Oct 30 13:29:17.101103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.100291 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-kube-api-access-4zp55\" (UniqueName: \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-kube-api-access-4zp55\") pod \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " Oct 30 13:29:17.101103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.100291 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-credential-socket" pod "1a94e7e5-3c0e-4f97-a16c-622e809fcd71" (UID: "1a94e7e5-3c0e-4f97-a16c-622e809fcd71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:17.101103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.100363 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-token\" (UniqueName: \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-token\") pod \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " Oct 30 13:29:17.101103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.100421 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istiod-ca-cert\") pod \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " Oct 30 13:29:17.101103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.100453 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-socket\") pod \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\" (UID: \"1a94e7e5-3c0e-4f97-a16c-622e809fcd71\") " Oct 30 13:29:17.101924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.101267 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.101924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.101787 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-socket" pod "1a94e7e5-3c0e-4f97-a16c-622e809fcd71" (UID: "1a94e7e5-3c0e-4f97-a16c-622e809fcd71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:17.102601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.102561 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istiod-ca-cert" pod "1a94e7e5-3c0e-4f97-a16c-622e809fcd71" (UID: "1a94e7e5-3c0e-4f97-a16c-622e809fcd71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:17.103114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.103071 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-data" pod "1a94e7e5-3c0e-4f97-a16c-622e809fcd71" (UID: "1a94e7e5-3c0e-4f97-a16c-622e809fcd71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:17.103538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.103493 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-certs" pod "1a94e7e5-3c0e-4f97-a16c-622e809fcd71" (UID: "1a94e7e5-3c0e-4f97-a16c-622e809fcd71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:17.106121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.106093 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-token" pod "1a94e7e5-3c0e-4f97-a16c-622e809fcd71" (UID: "1a94e7e5-3c0e-4f97-a16c-622e809fcd71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:17.106883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.106859 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-kube-api-access-4zp55" pod "1a94e7e5-3c0e-4f97-a16c-622e809fcd71" (UID: "1a94e7e5-3c0e-4f97-a16c-622e809fcd71"). InnerVolumeSpecName "kube-api-access-4zp55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:17.106988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.106959 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-podinfo" pod "1a94e7e5-3c0e-4f97-a16c-622e809fcd71" (UID: "1a94e7e5-3c0e-4f97-a16c-622e809fcd71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:29:17.107408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.107380 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-envoy" pod "1a94e7e5-3c0e-4f97-a16c-622e809fcd71" (UID: "1a94e7e5-3c0e-4f97-a16c-622e809fcd71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:17.202971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.202941 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.203042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.202993 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.203042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.203007 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.203042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.203020 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.203042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.203033 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zp55\" (UniqueName: \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-kube-api-access-4zp55\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.203185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.203047 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.203185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.203060 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.203185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.203073 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94e7e5-3c0e-4f97-a16c-622e809fcd71-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.220759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.220724 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb"] Oct 30 13:29:17.221923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.221897 5049 cpu_manager.go:414] "containerMap: removing container" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" Oct 30 13:29:17.222037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.221925 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" Oct 30 13:29:17.222037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.221957 5049 cpu_manager.go:414] "containerMap: removing container" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" Oct 30 13:29:17.222037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.221965 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" Oct 30 13:29:17.222037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.222031 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e11fca44-948f-4709-9632-0646ab90f3d5" containerName="revoke-keys" Oct 30 13:29:17.222205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.222039 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11fca44-948f-4709-9632-0646ab90f3d5" containerName="revoke-keys" Oct 30 13:29:17.222260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.222247 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" Oct 30 13:29:17.222308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.222277 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11fca44-948f-4709-9632-0646ab90f3d5" containerName="revoke-keys" Oct 30 13:29:17.227910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.227889 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.246995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.246973 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-9cbcbc-serving-cert\"" Oct 30 13:29:17.256102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.256076 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb"] Oct 30 13:29:17.305629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.305589 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.305629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.305640 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.305899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.305808 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/134b56fe-3e2a-4794-9eff-feb1120babb1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.305899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.305860 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcl6\" (UniqueName: \"kubernetes.io/projected/134b56fe-3e2a-4794-9eff-feb1120babb1-kube-api-access-vvcl6\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.399405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.399260 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q"] Oct 30 13:29:17.399853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.399826 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:17.404094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.404069 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-ptq4q"] Oct 30 13:29:17.407018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.406992 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/134b56fe-3e2a-4794-9eff-feb1120babb1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.407215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.407191 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcl6\" (UniqueName: \"kubernetes.io/projected/134b56fe-3e2a-4794-9eff-feb1120babb1-kube-api-access-vvcl6\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.407522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.407498 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.407617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.407533 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.407989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.407962 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/134b56fe-3e2a-4794-9eff-feb1120babb1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.410644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.410622 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.410734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.410674 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.430042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.430020 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcl6\" (UniqueName: \"kubernetes.io/projected/134b56fe-3e2a-4794-9eff-feb1120babb1-kube-api-access-vvcl6\") pod \"maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.542254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.542219 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:17.574711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.574682 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:17.856549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.856517 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb"] Oct 30 13:29:17.859225 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:29:17.859197 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134b56fe_3e2a_4794_9eff_feb1120babb1.slice/crio-5ba0985540dc94b90650a77e3b9734cf84970678e66fb470ba81d5971da826fa WatchSource:0}: Error finding container 5ba0985540dc94b90650a77e3b9734cf84970678e66fb470ba81d5971da826fa: Status 404 returned error can't find the container with id 5ba0985540dc94b90650a77e3b9734cf84970678e66fb470ba81d5971da826fa Oct 30 13:29:17.860737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.860715 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:29:17.906945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:17.906918 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:29:18.071618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:18.071583 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" event={"ID":"134b56fe-3e2a-4794-9eff-feb1120babb1","Type":"ContainerStarted","Data":"5ba0985540dc94b90650a77e3b9734cf84970678e66fb470ba81d5971da826fa"} Oct 30 13:29:19.083234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:19.083193 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" event={"ID":"134b56fe-3e2a-4794-9eff-feb1120babb1","Type":"ContainerStarted","Data":"37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9"} Oct 30 13:29:19.083757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:19.083263 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:19.127605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:19.127537 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" podStartSLOduration=1.707238268 podStartE2EDuration="2.127516977s" totalImagesPullingTime="420.278709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:19.122741656 +0000 UTC m=+2265.937936092" watchObservedRunningTime="2026-10-30 13:29:19.127516977 +0000 UTC m=+2265.942711400" Oct 30 13:29:19.329351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:19.326808 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:19.351391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:19.351270 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" path="/var/lib/kubelet/pods/1a94e7e5-3c0e-4f97-a16c-622e809fcd71/volumes" Oct 30 13:29:20.781314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:20.781275 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:20.969387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:20.969353 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:21.103760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.103726 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj"] Oct 30 13:29:21.104007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.103982 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" containerID="cri-o://fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d" gracePeriod=30 Oct 30 13:29:21.233344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.232772 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:21.485425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.485290 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:21.629144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.629104 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946"] Oct 30 13:29:21.630542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.630520 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a94e7e5-3c0e-4f97-a16c-622e809fcd71" containerName="istio-proxy" Oct 30 13:29:21.636355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.636316 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:21.650104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.650065 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:29:21.650978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.650958 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:29:21.655189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.655170 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946"] Oct 30 13:29:21.806203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.806165 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-376c89-hj946\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:21.806663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.806338 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-376c89-hj946\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:21.806663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.806398 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plz6g\" (UniqueName: \"kubernetes.io/projected/5993ce95-2514-401f-a815-e86f250b8d74-kube-api-access-plz6g\") pod \"maas-api-revoke-keys-e2e-inf-b-376c89-hj946\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:21.907984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.907942 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-376c89-hj946\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:21.908160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.908044 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-376c89-hj946\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:21.908160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.908067 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plz6g\" (UniqueName: \"kubernetes.io/projected/5993ce95-2514-401f-a815-e86f250b8d74-kube-api-access-plz6g\") pod \"maas-api-revoke-keys-e2e-inf-b-376c89-hj946\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:21.908863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.908834 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-376c89-hj946\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:21.910542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.910514 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-376c89-hj946\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:21.942665 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.942630 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plz6g\" (UniqueName: \"kubernetes.io/projected/5993ce95-2514-401f-a815-e86f250b8d74-kube-api-access-plz6g\") pod \"maas-api-revoke-keys-e2e-inf-b-376c89-hj946\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:21.957797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:21.957767 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:22.275722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:22.275692 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946"] Oct 30 13:29:22.278276 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:29:22.278250 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5993ce95_2514_401f_a815_e86f250b8d74.slice/crio-a00422fdbe47dc8cfca4441ddd1562810850bc96aba586c439a566d9e3b251cf WatchSource:0}: Error finding container a00422fdbe47dc8cfca4441ddd1562810850bc96aba586c439a566d9e3b251cf: Status 404 returned error can't find the container with id a00422fdbe47dc8cfca4441ddd1562810850bc96aba586c439a566d9e3b251cf Oct 30 13:29:22.771401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:22.771372 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:22.925514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:22.925477 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:23.136280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:23.136221 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" event={"ID":"5993ce95-2514-401f-a815-e86f250b8d74","Type":"ContainerStarted","Data":"b327057ff96e44768a4c7250c097f59aa7ae42bbca1e5a5ae7450e7cfdce97bf"} Oct 30 13:29:23.136589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:23.136563 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" event={"ID":"5993ce95-2514-401f-a815-e86f250b8d74","Type":"ContainerStarted","Data":"a00422fdbe47dc8cfca4441ddd1562810850bc96aba586c439a566d9e3b251cf"} Oct 30 13:29:23.205397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:23.199998 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" podStartSLOduration=2.199974798 podStartE2EDuration="2.199974798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:23.181583995 +0000 UTC m=+2269.996778419" watchObservedRunningTime="2026-10-30 13:29:23.199974798 +0000 UTC m=+2270.015169222" Oct 30 13:29:23.725398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:23.722659 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:24.149500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:24.149464 5049 generic.go:356] "Generic (PLEG): container finished" podID="5993ce95-2514-401f-a815-e86f250b8d74" containerID="b327057ff96e44768a4c7250c097f59aa7ae42bbca1e5a5ae7450e7cfdce97bf" exitCode=0 Oct 30 13:29:24.149947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:24.149528 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" event={"ID":"5993ce95-2514-401f-a815-e86f250b8d74","Type":"ContainerDied","Data":"b327057ff96e44768a4c7250c097f59aa7ae42bbca1e5a5ae7450e7cfdce97bf"} Oct 30 13:29:24.201793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:24.201764 5049 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:29:24.201951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:24.201813 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:29:25.100551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.100519 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:29:25.375838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.375812 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:25.473830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.473790 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5993ce95-2514-401f-a815-e86f250b8d74-kube-api-access-plz6g\" (UniqueName: \"kubernetes.io/projected/5993ce95-2514-401f-a815-e86f250b8d74-kube-api-access-plz6g\") pod \"5993ce95-2514-401f-a815-e86f250b8d74\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " Oct 30 13:29:25.474035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.473904 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-xks-ca\") pod \"5993ce95-2514-401f-a815-e86f250b8d74\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " Oct 30 13:29:25.474035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.473937 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-openshift-ca\") pod \"5993ce95-2514-401f-a815-e86f250b8d74\" (UID: \"5993ce95-2514-401f-a815-e86f250b8d74\") " Oct 30 13:29:25.474649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.474605 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-openshift-ca" pod "5993ce95-2514-401f-a815-e86f250b8d74" (UID: "5993ce95-2514-401f-a815-e86f250b8d74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:25.476249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.476222 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-xks-ca" pod "5993ce95-2514-401f-a815-e86f250b8d74" (UID: "5993ce95-2514-401f-a815-e86f250b8d74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:25.477305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.477274 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5993ce95-2514-401f-a815-e86f250b8d74-kube-api-access-plz6g" pod "5993ce95-2514-401f-a815-e86f250b8d74" (UID: "5993ce95-2514-401f-a815-e86f250b8d74"). InnerVolumeSpecName "kube-api-access-plz6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:25.576068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.576038 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:25.576068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.576066 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5993ce95-2514-401f-a815-e86f250b8d74-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:25.576263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.576077 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plz6g\" (UniqueName: \"kubernetes.io/projected/5993ce95-2514-401f-a815-e86f250b8d74-kube-api-access-plz6g\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:25.633584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:25.633543 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:26.172442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.172414 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" Oct 30 13:29:26.172601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.172434 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946" event={"ID":"5993ce95-2514-401f-a815-e86f250b8d74","Type":"ContainerDied","Data":"a00422fdbe47dc8cfca4441ddd1562810850bc96aba586c439a566d9e3b251cf"} Oct 30 13:29:26.172601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.172488 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a00422fdbe47dc8cfca4441ddd1562810850bc96aba586c439a566d9e3b251cf" Oct 30 13:29:26.404235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.404209 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:29:26.491198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491168 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-socket\") pod \"30514d49-dc56-4c0a-bbfd-775dc42dc550\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " Oct 30 13:29:26.491420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491209 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-certs\") pod \"30514d49-dc56-4c0a-bbfd-775dc42dc550\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " Oct 30 13:29:26.491420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491334 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30514d49-dc56-4c0a-bbfd-775dc42dc550-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30514d49-dc56-4c0a-bbfd-775dc42dc550-istiod-ca-cert\") pod \"30514d49-dc56-4c0a-bbfd-775dc42dc550\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " Oct 30 13:29:26.491420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491395 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-credential-socket\") pod \"30514d49-dc56-4c0a-bbfd-775dc42dc550\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " Oct 30 13:29:26.491622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491458 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-token\" (UniqueName: \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-token\") pod \"30514d49-dc56-4c0a-bbfd-775dc42dc550\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " Oct 30 13:29:26.491622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491510 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-envoy\") pod \"30514d49-dc56-4c0a-bbfd-775dc42dc550\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " Oct 30 13:29:26.491622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491576 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-certs" pod "30514d49-dc56-4c0a-bbfd-775dc42dc550" (UID: "30514d49-dc56-4c0a-bbfd-775dc42dc550"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:26.491793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491636 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-data\") pod \"30514d49-dc56-4c0a-bbfd-775dc42dc550\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " Oct 30 13:29:26.491793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491678 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-kube-api-access-tzh5c\" (UniqueName: \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-kube-api-access-tzh5c\") pod \"30514d49-dc56-4c0a-bbfd-775dc42dc550\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " Oct 30 13:29:26.491793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491767 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-podinfo\") pod \"30514d49-dc56-4c0a-bbfd-775dc42dc550\" (UID: \"30514d49-dc56-4c0a-bbfd-775dc42dc550\") " Oct 30 13:29:26.491949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491806 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30514d49-dc56-4c0a-bbfd-775dc42dc550-istiod-ca-cert" pod "30514d49-dc56-4c0a-bbfd-775dc42dc550" (UID: "30514d49-dc56-4c0a-bbfd-775dc42dc550"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:26.491949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.491815 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-credential-socket" pod "30514d49-dc56-4c0a-bbfd-775dc42dc550" (UID: "30514d49-dc56-4c0a-bbfd-775dc42dc550"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:26.492357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.492314 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-socket" pod "30514d49-dc56-4c0a-bbfd-775dc42dc550" (UID: "30514d49-dc56-4c0a-bbfd-775dc42dc550"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:26.492357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.492345 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-data" pod "30514d49-dc56-4c0a-bbfd-775dc42dc550" (UID: "30514d49-dc56-4c0a-bbfd-775dc42dc550"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:26.492520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.492397 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:26.492520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.492418 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:26.492520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.492432 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30514d49-dc56-4c0a-bbfd-775dc42dc550-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:26.494722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.494693 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-token" pod "30514d49-dc56-4c0a-bbfd-775dc42dc550" (UID: "30514d49-dc56-4c0a-bbfd-775dc42dc550"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:26.494857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.494832 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-kube-api-access-tzh5c" pod "30514d49-dc56-4c0a-bbfd-775dc42dc550" (UID: "30514d49-dc56-4c0a-bbfd-775dc42dc550"). InnerVolumeSpecName "kube-api-access-tzh5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:26.494928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.494902 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-podinfo" pod "30514d49-dc56-4c0a-bbfd-775dc42dc550" (UID: "30514d49-dc56-4c0a-bbfd-775dc42dc550"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:29:26.495745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.495713 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-envoy" pod "30514d49-dc56-4c0a-bbfd-775dc42dc550" (UID: "30514d49-dc56-4c0a-bbfd-775dc42dc550"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:26.593721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.593685 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:26.593721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.593714 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:26.593721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.593723 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:26.593963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.593732 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzh5c\" (UniqueName: \"kubernetes.io/projected/30514d49-dc56-4c0a-bbfd-775dc42dc550-kube-api-access-tzh5c\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:26.593963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.593742 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30514d49-dc56-4c0a-bbfd-775dc42dc550-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:26.593963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:26.593750 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30514d49-dc56-4c0a-bbfd-775dc42dc550-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:26.678986 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:29:26.678905 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:29:27.182577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.182546 5049 generic.go:356] "Generic (PLEG): container finished" podID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerID="fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d" exitCode=0 Oct 30 13:29:27.182761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.182592 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" Oct 30 13:29:27.182761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.182619 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" event={"ID":"30514d49-dc56-4c0a-bbfd-775dc42dc550","Type":"ContainerDied","Data":"fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d"} Oct 30 13:29:27.182761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.182659 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj" event={"ID":"30514d49-dc56-4c0a-bbfd-775dc42dc550","Type":"ContainerDied","Data":"1b9ed3ce5c262f2cd7f9a8a8778323d2e6923691aae52ed70adcb817c8c0e619"} Oct 30 13:29:27.182761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.182679 5049 scope.go:122] "RemoveContainer" containerID="fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d" Oct 30 13:29:27.196408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.196390 5049 scope.go:122] "RemoveContainer" containerID="fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d" Oct 30 13:29:27.196737 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:29:27.196715 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d\": container with ID starting with fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d not found: ID does not exist" containerID="fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d" Oct 30 13:29:27.196821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.196746 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d"} err="failed to get container status \"fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d\": rpc error: code = NotFound desc = could not find container \"fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d\": container with ID starting with fb3e8046dd1d5697552c234bdff320f92943d573f4c9eda636fe20a2ee709d0d not found: ID does not exist" Oct 30 13:29:27.235189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.235159 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj"] Oct 30 13:29:27.242701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.242677 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-j9bbj"] Oct 30 13:29:27.341729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.341695 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" path="/var/lib/kubelet/pods/30514d49-dc56-4c0a-bbfd-775dc42dc550/volumes" Oct 30 13:29:27.478140 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:27.478028 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:28.450216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:28.450178 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:29.271995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:29.271953 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:31.200743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:31.200699 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:32.220268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:32.220233 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:32.379096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:32.379063 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:33.216131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:33.216026 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:35.208359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:35.208303 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:37.199591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:37.199549 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:39.156891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.156837 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:39.263084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.263045 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn"] Oct 30 13:29:39.263879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.263861 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5993ce95-2514-401f-a815-e86f250b8d74" containerName="revoke-keys" Oct 30 13:29:39.263962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.263879 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5993ce95-2514-401f-a815-e86f250b8d74" containerName="revoke-keys" Oct 30 13:29:39.263962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.263899 5049 cpu_manager.go:414] "containerMap: removing container" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" Oct 30 13:29:39.263962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.263905 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" Oct 30 13:29:39.264064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.264023 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5993ce95-2514-401f-a815-e86f250b8d74" containerName="revoke-keys" Oct 30 13:29:39.264064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.264037 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="30514d49-dc56-4c0a-bbfd-775dc42dc550" containerName="istio-proxy" Oct 30 13:29:39.269390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.269372 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.284438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.284409 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:29:39.294123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.294090 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:29:39.300039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.300001 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn"] Oct 30 13:29:39.443922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.443832 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2f4fda1-z86bn\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.444064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.444006 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8xn\" (UniqueName: \"kubernetes.io/projected/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-kube-api-access-fm8xn\") pod \"maas-api-revoke-sub-b2f4fda1-z86bn\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.444128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.444114 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2f4fda1-z86bn\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.546256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.546213 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8xn\" (UniqueName: \"kubernetes.io/projected/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-kube-api-access-fm8xn\") pod \"maas-api-revoke-sub-b2f4fda1-z86bn\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.546465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.546365 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2f4fda1-z86bn\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.546624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.546606 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2f4fda1-z86bn\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.547467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.547438 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2f4fda1-z86bn\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.548907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.548883 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2f4fda1-z86bn\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.568298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.568275 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8xn\" (UniqueName: \"kubernetes.io/projected/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-kube-api-access-fm8xn\") pod \"maas-api-revoke-sub-b2f4fda1-z86bn\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.583552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.583524 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:39.907041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:39.907012 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn"] Oct 30 13:29:39.910135 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:29:39.910100 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf8ebcc_dc89_4279_97ca_e0d5d052cb18.slice/crio-aaa736d2fe3ca26a1e70b1bc8b229fae503b31293a0edb40241a180f6d177a6f WatchSource:0}: Error finding container aaa736d2fe3ca26a1e70b1bc8b229fae503b31293a0edb40241a180f6d177a6f: Status 404 returned error can't find the container with id aaa736d2fe3ca26a1e70b1bc8b229fae503b31293a0edb40241a180f6d177a6f Oct 30 13:29:40.344916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:40.344871 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" event={"ID":"daf8ebcc-dc89-4279-97ca-e0d5d052cb18","Type":"ContainerStarted","Data":"2841ee77a5ac8f0e9754184867efd7d3fec032f9748362cecc48df829b575806"} Oct 30 13:29:40.345307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:40.344937 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" event={"ID":"daf8ebcc-dc89-4279-97ca-e0d5d052cb18","Type":"ContainerStarted","Data":"aaa736d2fe3ca26a1e70b1bc8b229fae503b31293a0edb40241a180f6d177a6f"} Oct 30 13:29:40.382512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:40.382451 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" podStartSLOduration=1.3824314260000001 podStartE2EDuration="1.382431426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:40.381093868 +0000 UTC m=+2287.196288291" watchObservedRunningTime="2026-10-30 13:29:40.382431426 +0000 UTC m=+2287.197625855" Oct 30 13:29:40.960873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:40.960785 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:41.355721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:41.355682 5049 generic.go:356] "Generic (PLEG): container finished" podID="daf8ebcc-dc89-4279-97ca-e0d5d052cb18" containerID="2841ee77a5ac8f0e9754184867efd7d3fec032f9748362cecc48df829b575806" exitCode=0 Oct 30 13:29:41.356099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:41.355749 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" event={"ID":"daf8ebcc-dc89-4279-97ca-e0d5d052cb18","Type":"ContainerDied","Data":"2841ee77a5ac8f0e9754184867efd7d3fec032f9748362cecc48df829b575806"} Oct 30 13:29:42.581858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.581826 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:42.693392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.692060 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-openshift-ca\") pod \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " Oct 30 13:29:42.693392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.692145 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-xks-ca\") pod \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " Oct 30 13:29:42.693392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.692207 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-kube-api-access-fm8xn\" (UniqueName: \"kubernetes.io/projected/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-kube-api-access-fm8xn\") pod \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\" (UID: \"daf8ebcc-dc89-4279-97ca-e0d5d052cb18\") " Oct 30 13:29:42.693392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.692432 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-openshift-ca" pod "daf8ebcc-dc89-4279-97ca-e0d5d052cb18" (UID: "daf8ebcc-dc89-4279-97ca-e0d5d052cb18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:42.693392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.692916 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:42.695197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.695171 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-kube-api-access-fm8xn" pod "daf8ebcc-dc89-4279-97ca-e0d5d052cb18" (UID: "daf8ebcc-dc89-4279-97ca-e0d5d052cb18"). InnerVolumeSpecName "kube-api-access-fm8xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:42.697978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.697950 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-xks-ca" pod "daf8ebcc-dc89-4279-97ca-e0d5d052cb18" (UID: "daf8ebcc-dc89-4279-97ca-e0d5d052cb18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:42.794396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.794366 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:42.794396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.794392 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm8xn\" (UniqueName: \"kubernetes.io/projected/daf8ebcc-dc89-4279-97ca-e0d5d052cb18-kube-api-access-fm8xn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:42.980684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:42.980591 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:43.374391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:43.374356 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" Oct 30 13:29:43.374391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:43.374369 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn" event={"ID":"daf8ebcc-dc89-4279-97ca-e0d5d052cb18","Type":"ContainerDied","Data":"aaa736d2fe3ca26a1e70b1bc8b229fae503b31293a0edb40241a180f6d177a6f"} Oct 30 13:29:43.374590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:43.374416 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaa736d2fe3ca26a1e70b1bc8b229fae503b31293a0edb40241a180f6d177a6f" Oct 30 13:29:44.811635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:44.811546 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:45.000481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:45.000452 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:46.863143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:46.863086 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:48.671890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:48.671855 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:49.651093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.651039 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5"] Oct 30 13:29:49.652229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.652207 5049 cpu_manager.go:414] "containerMap: removing container" podUID="daf8ebcc-dc89-4279-97ca-e0d5d052cb18" containerName="revoke-keys" Oct 30 13:29:49.652292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.652233 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf8ebcc-dc89-4279-97ca-e0d5d052cb18" containerName="revoke-keys" Oct 30 13:29:49.652520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.652506 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf8ebcc-dc89-4279-97ca-e0d5d052cb18" containerName="revoke-keys" Oct 30 13:29:49.658449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.658426 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:49.674754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.674725 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5"] Oct 30 13:29:49.679670 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.679648 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:29:49.679846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.679731 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:29:49.812306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.812270 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cbda8b-wv8z5\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:49.812530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.812334 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbpb8\" (UniqueName: \"kubernetes.io/projected/df37af6c-fd30-44c3-bc82-2b4166f0825d-kube-api-access-tbpb8\") pod \"maas-api-revoke-sub-33cbda8b-wv8z5\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:49.812530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.812471 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cbda8b-wv8z5\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:49.913962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.913873 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cbda8b-wv8z5\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:49.913962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.913925 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbpb8\" (UniqueName: \"kubernetes.io/projected/df37af6c-fd30-44c3-bc82-2b4166f0825d-kube-api-access-tbpb8\") pod \"maas-api-revoke-sub-33cbda8b-wv8z5\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:49.914201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.913982 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cbda8b-wv8z5\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:49.914754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.914728 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cbda8b-wv8z5\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:49.916275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.916255 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cbda8b-wv8z5\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:49.937597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.937569 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbpb8\" (UniqueName: \"kubernetes.io/projected/df37af6c-fd30-44c3-bc82-2b4166f0825d-kube-api-access-tbpb8\") pod \"maas-api-revoke-sub-33cbda8b-wv8z5\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:49.973296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:49.973270 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:50.282444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:50.282415 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5"] Oct 30 13:29:50.285000 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:29:50.284973 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf37af6c_fd30_44c3_bc82_2b4166f0825d.slice/crio-b20116d31fd69fc16703c8c44617a0e845ccfc9b0219a344fae75ecda107b5d1 WatchSource:0}: Error finding container b20116d31fd69fc16703c8c44617a0e845ccfc9b0219a344fae75ecda107b5d1: Status 404 returned error can't find the container with id b20116d31fd69fc16703c8c44617a0e845ccfc9b0219a344fae75ecda107b5d1 Oct 30 13:29:50.441606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:50.441287 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:50.473846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:50.473792 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" event={"ID":"df37af6c-fd30-44c3-bc82-2b4166f0825d","Type":"ContainerStarted","Data":"b20116d31fd69fc16703c8c44617a0e845ccfc9b0219a344fae75ecda107b5d1"} Oct 30 13:29:51.484288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:51.484200 5049 generic.go:356] "Generic (PLEG): container finished" podID="df37af6c-fd30-44c3-bc82-2b4166f0825d" containerID="5ae47b44ca7e275fcc894da5bef348d455c2cc968ccde642d550471f65815bde" exitCode=0 Oct 30 13:29:51.484716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:51.484278 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" event={"ID":"df37af6c-fd30-44c3-bc82-2b4166f0825d","Type":"ContainerDied","Data":"5ae47b44ca7e275fcc894da5bef348d455c2cc968ccde642d550471f65815bde"} Oct 30 13:29:52.014465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.014429 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:52.756812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.756778 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:52.779867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.779838 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:52.905282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.905250 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb"] Oct 30 13:29:52.910477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.910451 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c7ba3-md8hb"] Oct 30 13:29:52.942437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.942404 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:52.974026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.973997 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-xks-ca\") pod \"df37af6c-fd30-44c3-bc82-2b4166f0825d\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " Oct 30 13:29:52.974198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.974064 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df37af6c-fd30-44c3-bc82-2b4166f0825d-kube-api-access-tbpb8\" (UniqueName: \"kubernetes.io/projected/df37af6c-fd30-44c3-bc82-2b4166f0825d-kube-api-access-tbpb8\") pod \"df37af6c-fd30-44c3-bc82-2b4166f0825d\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " Oct 30 13:29:52.974198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.974165 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-openshift-ca\") pod \"df37af6c-fd30-44c3-bc82-2b4166f0825d\" (UID: \"df37af6c-fd30-44c3-bc82-2b4166f0825d\") " Oct 30 13:29:52.974577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.974545 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-openshift-ca" pod "df37af6c-fd30-44c3-bc82-2b4166f0825d" (UID: "df37af6c-fd30-44c3-bc82-2b4166f0825d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:52.975012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.974989 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:52.976845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.976816 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-xks-ca" pod "df37af6c-fd30-44c3-bc82-2b4166f0825d" (UID: "df37af6c-fd30-44c3-bc82-2b4166f0825d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:52.976967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:52.976858 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df37af6c-fd30-44c3-bc82-2b4166f0825d-kube-api-access-tbpb8" pod "df37af6c-fd30-44c3-bc82-2b4166f0825d" (UID: "df37af6c-fd30-44c3-bc82-2b4166f0825d"). InnerVolumeSpecName "kube-api-access-tbpb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:53.075997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:53.075965 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df37af6c-fd30-44c3-bc82-2b4166f0825d-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:53.075997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:53.075994 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbpb8\" (UniqueName: \"kubernetes.io/projected/df37af6c-fd30-44c3-bc82-2b4166f0825d-kube-api-access-tbpb8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:29:53.117017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:53.116986 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:53.342700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:53.342622 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e11fca44-948f-4709-9632-0646ab90f3d5" path="/var/lib/kubelet/pods/e11fca44-948f-4709-9632-0646ab90f3d5/volumes" Oct 30 13:29:53.503266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:53.503233 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" event={"ID":"df37af6c-fd30-44c3-bc82-2b4166f0825d","Type":"ContainerDied","Data":"b20116d31fd69fc16703c8c44617a0e845ccfc9b0219a344fae75ecda107b5d1"} Oct 30 13:29:53.503266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:53.503254 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5" Oct 30 13:29:53.503528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:53.503279 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b20116d31fd69fc16703c8c44617a0e845ccfc9b0219a344fae75ecda107b5d1" Oct 30 13:29:53.520808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:53.520776 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:55.228399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:55.228367 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:56.680943 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:29:56.680902 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:29:56.966399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:56.965336 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:58.969280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:58.969223 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:59.565550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:29:59.565517 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:00.028303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.028219 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:00.148284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.148254 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk"] Oct 30 13:30:00.149558 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.149531 5049 cpu_manager.go:414] "containerMap: removing container" podUID="df37af6c-fd30-44c3-bc82-2b4166f0825d" containerName="revoke-keys" Oct 30 13:30:00.149736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.149718 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df37af6c-fd30-44c3-bc82-2b4166f0825d" containerName="revoke-keys" Oct 30 13:30:00.150114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.150090 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="df37af6c-fd30-44c3-bc82-2b4166f0825d" containerName="revoke-keys" Oct 30 13:30:00.154010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.153984 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" Oct 30 13:30:00.157537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.157513 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7"] Oct 30 13:30:00.162444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.162154 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" Oct 30 13:30:00.182246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.182218 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:30:00.190330 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.190002 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b"] Oct 30 13:30:00.204843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.204806 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fksvv\" (UniqueName: \"kubernetes.io/projected/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-kube-api-access-fksvv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk\" (UID: \"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" Oct 30 13:30:00.205200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.205176 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk\" (UID: \"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" Oct 30 13:30:00.210908 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.210866 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk"] Oct 30 13:30:00.211035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.210930 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7"] Oct 30 13:30:00.211035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.210976 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z"] Oct 30 13:30:00.211751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.211732 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" Oct 30 13:30:00.223302 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.222518 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b"] Oct 30 13:30:00.223302 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.222626 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" Oct 30 13:30:00.251685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.251651 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z"] Oct 30 13:30:00.251887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.251725 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj"] Oct 30 13:30:00.257786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.257760 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x"] Oct 30 13:30:00.258045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.258022 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" Oct 30 13:30:00.264559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.264440 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" Oct 30 13:30:00.280023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.279951 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj"] Oct 30 13:30:00.286991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.286948 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x"] Oct 30 13:30:00.292959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.292933 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt"] Oct 30 13:30:00.298622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.298601 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.314772 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8g7m\" (UniqueName: \"kubernetes.io/projected/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-kube-api-access-b8g7m\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7\" (UID: \"7943ae21-45c1-466b-a9ff-9bfc8dbfe485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.314813 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4633f32-74bb-4ff0-ba96-7baef8fd130f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b\" (UID: \"c4633f32-74bb-4ff0-ba96-7baef8fd130f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.314928 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fksvv\" (UniqueName: \"kubernetes.io/projected/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-kube-api-access-fksvv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk\" (UID: \"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.315004 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fh7g\" (UniqueName: \"kubernetes.io/projected/8e317c46-94d2-4900-b307-6ac09a1d832d-kube-api-access-4fh7g\") pod \"maas-api-key-cleanup-29889450-t4jnj\" (UID: \"8e317c46-94d2-4900-b307-6ac09a1d832d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.315094 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88cxz\" (UniqueName: \"kubernetes.io/projected/c4633f32-74bb-4ff0-ba96-7baef8fd130f-kube-api-access-88cxz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b\" (UID: \"c4633f32-74bb-4ff0-ba96-7baef8fd130f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.315124 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk\" (UID: \"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.315149 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7\" (UID: \"7943ae21-45c1-466b-a9ff-9bfc8dbfe485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.315194 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z\" (UID: \"978b344c-24b0-4fc2-a8f0-7c72d7510ceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.315387 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e317c46-94d2-4900-b307-6ac09a1d832d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-t4jnj\" (UID: \"8e317c46-94d2-4900-b307-6ac09a1d832d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.315454 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8fb6\" (UniqueName: \"kubernetes.io/projected/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-kube-api-access-c8fb6\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z\" (UID: \"978b344c-24b0-4fc2-a8f0-7c72d7510ceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" Oct 30 13:30:00.316854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.316558 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk\" (UID: \"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" Oct 30 13:30:00.334071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.333575 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt"] Oct 30 13:30:00.352235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.352210 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fksvv\" (UniqueName: \"kubernetes.io/projected/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-kube-api-access-fksvv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk\" (UID: \"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" Oct 30 13:30:00.416374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.416305 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88cxz\" (UniqueName: \"kubernetes.io/projected/c4633f32-74bb-4ff0-ba96-7baef8fd130f-kube-api-access-88cxz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b\" (UID: \"c4633f32-74bb-4ff0-ba96-7baef8fd130f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" Oct 30 13:30:00.416576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.416386 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7\" (UID: \"7943ae21-45c1-466b-a9ff-9bfc8dbfe485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" Oct 30 13:30:00.416576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.416426 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z\" (UID: \"978b344c-24b0-4fc2-a8f0-7c72d7510ceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" Oct 30 13:30:00.416708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.416611 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvdq\" (UniqueName: \"kubernetes.io/projected/4b88874a-3d0a-4029-8581-7c76a39201a2-kube-api-access-nxvdq\") pod \"maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt\" (UID: \"4b88874a-3d0a-4029-8581-7c76a39201a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" Oct 30 13:30:00.416770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.416748 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b88874a-3d0a-4029-8581-7c76a39201a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt\" (UID: \"4b88874a-3d0a-4029-8581-7c76a39201a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" Oct 30 13:30:00.416929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.416906 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8dh5\" (UniqueName: \"kubernetes.io/projected/cc49db5d-04ab-4ca6-b0c2-9094909526b2-kube-api-access-k8dh5\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x\" (UID: \"cc49db5d-04ab-4ca6-b0c2-9094909526b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" Oct 30 13:30:00.417026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.417006 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc49db5d-04ab-4ca6-b0c2-9094909526b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x\" (UID: \"cc49db5d-04ab-4ca6-b0c2-9094909526b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" Oct 30 13:30:00.417089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.417068 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e317c46-94d2-4900-b307-6ac09a1d832d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-t4jnj\" (UID: \"8e317c46-94d2-4900-b307-6ac09a1d832d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" Oct 30 13:30:00.417149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.417135 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8fb6\" (UniqueName: \"kubernetes.io/projected/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-kube-api-access-c8fb6\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z\" (UID: \"978b344c-24b0-4fc2-a8f0-7c72d7510ceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" Oct 30 13:30:00.417209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.417197 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8g7m\" (UniqueName: \"kubernetes.io/projected/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-kube-api-access-b8g7m\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7\" (UID: \"7943ae21-45c1-466b-a9ff-9bfc8dbfe485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" Oct 30 13:30:00.417266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.417221 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4633f32-74bb-4ff0-ba96-7baef8fd130f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b\" (UID: \"c4633f32-74bb-4ff0-ba96-7baef8fd130f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" Oct 30 13:30:00.417266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.417235 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7\" (UID: \"7943ae21-45c1-466b-a9ff-9bfc8dbfe485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" Oct 30 13:30:00.417266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.417252 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z\" (UID: \"978b344c-24b0-4fc2-a8f0-7c72d7510ceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" Oct 30 13:30:00.417519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.417373 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fh7g\" (UniqueName: \"kubernetes.io/projected/8e317c46-94d2-4900-b307-6ac09a1d832d-kube-api-access-4fh7g\") pod \"maas-api-key-cleanup-29889450-t4jnj\" (UID: \"8e317c46-94d2-4900-b307-6ac09a1d832d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" Oct 30 13:30:00.417807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.417783 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e317c46-94d2-4900-b307-6ac09a1d832d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-t4jnj\" (UID: \"8e317c46-94d2-4900-b307-6ac09a1d832d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" Oct 30 13:30:00.417935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.417918 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4633f32-74bb-4ff0-ba96-7baef8fd130f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b\" (UID: \"c4633f32-74bb-4ff0-ba96-7baef8fd130f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" Oct 30 13:30:00.461098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.461061 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8g7m\" (UniqueName: \"kubernetes.io/projected/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-kube-api-access-b8g7m\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7\" (UID: \"7943ae21-45c1-466b-a9ff-9bfc8dbfe485\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" Oct 30 13:30:00.461273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.461243 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8fb6\" (UniqueName: \"kubernetes.io/projected/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-kube-api-access-c8fb6\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z\" (UID: \"978b344c-24b0-4fc2-a8f0-7c72d7510ceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" Oct 30 13:30:00.464308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.464280 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fh7g\" (UniqueName: \"kubernetes.io/projected/8e317c46-94d2-4900-b307-6ac09a1d832d-kube-api-access-4fh7g\") pod \"maas-api-key-cleanup-29889450-t4jnj\" (UID: \"8e317c46-94d2-4900-b307-6ac09a1d832d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" Oct 30 13:30:00.464469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.464283 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88cxz\" (UniqueName: \"kubernetes.io/projected/c4633f32-74bb-4ff0-ba96-7baef8fd130f-kube-api-access-88cxz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b\" (UID: \"c4633f32-74bb-4ff0-ba96-7baef8fd130f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" Oct 30 13:30:00.468926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.468905 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" Oct 30 13:30:00.482572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.482546 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" Oct 30 13:30:00.521762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.519302 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b88874a-3d0a-4029-8581-7c76a39201a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt\" (UID: \"4b88874a-3d0a-4029-8581-7c76a39201a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" Oct 30 13:30:00.521762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.519404 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dh5\" (UniqueName: \"kubernetes.io/projected/cc49db5d-04ab-4ca6-b0c2-9094909526b2-kube-api-access-k8dh5\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x\" (UID: \"cc49db5d-04ab-4ca6-b0c2-9094909526b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" Oct 30 13:30:00.521762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.519474 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc49db5d-04ab-4ca6-b0c2-9094909526b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x\" (UID: \"cc49db5d-04ab-4ca6-b0c2-9094909526b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" Oct 30 13:30:00.521762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.519751 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvdq\" (UniqueName: \"kubernetes.io/projected/4b88874a-3d0a-4029-8581-7c76a39201a2-kube-api-access-nxvdq\") pod \"maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt\" (UID: \"4b88874a-3d0a-4029-8581-7c76a39201a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" Oct 30 13:30:00.521762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.520834 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b88874a-3d0a-4029-8581-7c76a39201a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt\" (UID: \"4b88874a-3d0a-4029-8581-7c76a39201a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" Oct 30 13:30:00.521762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.521717 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc49db5d-04ab-4ca6-b0c2-9094909526b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x\" (UID: \"cc49db5d-04ab-4ca6-b0c2-9094909526b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" Oct 30 13:30:00.544426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.544402 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvdq\" (UniqueName: \"kubernetes.io/projected/4b88874a-3d0a-4029-8581-7c76a39201a2-kube-api-access-nxvdq\") pod \"maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt\" (UID: \"4b88874a-3d0a-4029-8581-7c76a39201a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" Oct 30 13:30:00.570654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.570623 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" Oct 30 13:30:00.573657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.573626 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" Oct 30 13:30:00.574072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.574041 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8dh5\" (UniqueName: \"kubernetes.io/projected/cc49db5d-04ab-4ca6-b0c2-9094909526b2-kube-api-access-k8dh5\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x\" (UID: \"cc49db5d-04ab-4ca6-b0c2-9094909526b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" Oct 30 13:30:00.597247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.597041 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" Oct 30 13:30:00.614238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.614204 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" Oct 30 13:30:00.683505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.682380 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" Oct 30 13:30:00.804680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.804645 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:00.965627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:00.965539 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:01.143890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.143768 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:01.145946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.144728 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z"] Oct 30 13:30:01.146063 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:01.146027 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978b344c_24b0_4fc2_a8f0_7c72d7510ceb.slice/crio-c3491c833adba7da4e80ba44ec722d646f5534c3b00b33f56b12ae3bac2f80c6 WatchSource:0}: Error finding container c3491c833adba7da4e80ba44ec722d646f5534c3b00b33f56b12ae3bac2f80c6: Status 404 returned error can't find the container with id c3491c833adba7da4e80ba44ec722d646f5534c3b00b33f56b12ae3bac2f80c6 Oct 30 13:30:01.196561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.189806 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj"] Oct 30 13:30:01.208566 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:01.208535 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e317c46_94d2_4900_b307_6ac09a1d832d.slice/crio-4e5660dce5ccb80ec386d4fb19d3c5e039d68eb7788d11ff85e293f026e568cb WatchSource:0}: Error finding container 4e5660dce5ccb80ec386d4fb19d3c5e039d68eb7788d11ff85e293f026e568cb: Status 404 returned error can't find the container with id 4e5660dce5ccb80ec386d4fb19d3c5e039d68eb7788d11ff85e293f026e568cb Oct 30 13:30:01.219442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.218919 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b"] Oct 30 13:30:01.229583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.229557 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7"] Oct 30 13:30:01.239797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.238439 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk"] Oct 30 13:30:01.244309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.244287 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt"] Oct 30 13:30:01.278185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.278160 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x"] Oct 30 13:30:01.285181 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:01.285149 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc49db5d_04ab_4ca6_b0c2_9094909526b2.slice/crio-7ff8ed8e117a70d0ae12d563ab023c3216c599528f240d15cdee0918e2d70e14 WatchSource:0}: Error finding container 7ff8ed8e117a70d0ae12d563ab023c3216c599528f240d15cdee0918e2d70e14: Status 404 returned error can't find the container with id 7ff8ed8e117a70d0ae12d563ab023c3216c599528f240d15cdee0918e2d70e14 Oct 30 13:30:01.363104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.363028 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg"] Oct 30 13:30:01.363250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.363211 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" containerID="cri-o://bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a" gracePeriod=30 Oct 30 13:30:01.434781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.434752 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:01.602057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.602020 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" event={"ID":"8e317c46-94d2-4900-b307-6ac09a1d832d","Type":"ContainerStarted","Data":"4e5660dce5ccb80ec386d4fb19d3c5e039d68eb7788d11ff85e293f026e568cb"} Oct 30 13:30:01.609097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.609062 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" event={"ID":"4b88874a-3d0a-4029-8581-7c76a39201a2","Type":"ContainerStarted","Data":"4613ad0658821d33b845f709e491a27315973d5c3456200cbb4e279bc58b43d2"} Oct 30 13:30:01.611047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.611017 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" event={"ID":"c4633f32-74bb-4ff0-ba96-7baef8fd130f","Type":"ContainerStarted","Data":"7f650d5b1fde7665dbfdc1c7fd5041ef1745858ad4e4484cf08815c5a43079b4"} Oct 30 13:30:01.613182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.613116 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" event={"ID":"7943ae21-45c1-466b-a9ff-9bfc8dbfe485","Type":"ContainerStarted","Data":"e3e265ed8d27c935a8991603eb759623adad2819f6bb5e21600e68a47945b824"} Oct 30 13:30:01.614893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.614869 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" event={"ID":"cc49db5d-04ab-4ca6-b0c2-9094909526b2","Type":"ContainerStarted","Data":"7ff8ed8e117a70d0ae12d563ab023c3216c599528f240d15cdee0918e2d70e14"} Oct 30 13:30:01.616760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.616738 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" event={"ID":"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2","Type":"ContainerStarted","Data":"b99ebe8d45f36bacff4f071a028f6ad94d6195c3080d0bcdb78ad9ff4b04981d"} Oct 30 13:30:01.618769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.618748 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" event={"ID":"978b344c-24b0-4fc2-a8f0-7c72d7510ceb","Type":"ContainerStarted","Data":"c3491c833adba7da4e80ba44ec722d646f5534c3b00b33f56b12ae3bac2f80c6"} Oct 30 13:30:01.680403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:01.668212 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:02.098993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.098958 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k"] Oct 30 13:30:02.175737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.175662 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:02.337832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.337798 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:02.629706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.629664 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" event={"ID":"cc49db5d-04ab-4ca6-b0c2-9094909526b2","Type":"ContainerStarted","Data":"2fc0e96df1f8557f09694f45f0d14be9add21cfa66f4f7affa46bc20dd808293"} Oct 30 13:30:02.631481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.631454 5049 generic.go:356] "Generic (PLEG): container finished" podID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerID="c02ebf97c4c02f1e76ae7c7cf6c6d8460c6cfd406a2228447914893731f90851" exitCode=60 Oct 30 13:30:02.631618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.631538 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" event={"ID":"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2","Type":"ContainerDied","Data":"c02ebf97c4c02f1e76ae7c7cf6c6d8460c6cfd406a2228447914893731f90851"} Oct 30 13:30:02.631822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.631803 5049 scope.go:122] "RemoveContainer" containerID="c02ebf97c4c02f1e76ae7c7cf6c6d8460c6cfd406a2228447914893731f90851" Oct 30 13:30:02.635047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.635021 5049 generic.go:356] "Generic (PLEG): container finished" podID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerID="544094c40c0d9746195cd2c22a3ddf8f513b54a43f8b0193f03c4f402d69abc5" exitCode=60 Oct 30 13:30:02.635153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.635058 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" event={"ID":"978b344c-24b0-4fc2-a8f0-7c72d7510ceb","Type":"ContainerDied","Data":"544094c40c0d9746195cd2c22a3ddf8f513b54a43f8b0193f03c4f402d69abc5"} Oct 30 13:30:02.635501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.635391 5049 scope.go:122] "RemoveContainer" containerID="544094c40c0d9746195cd2c22a3ddf8f513b54a43f8b0193f03c4f402d69abc5" Oct 30 13:30:02.637214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.637187 5049 generic.go:356] "Generic (PLEG): container finished" podID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerID="01284edf2cef1cd451e6cc38054c08637a369a95abef99439d078a803ce31989" exitCode=60 Oct 30 13:30:02.637331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.637258 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" event={"ID":"8e317c46-94d2-4900-b307-6ac09a1d832d","Type":"ContainerDied","Data":"01284edf2cef1cd451e6cc38054c08637a369a95abef99439d078a803ce31989"} Oct 30 13:30:02.637698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.637668 5049 scope.go:122] "RemoveContainer" containerID="01284edf2cef1cd451e6cc38054c08637a369a95abef99439d078a803ce31989" Oct 30 13:30:02.640588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.640570 5049 generic.go:356] "Generic (PLEG): container finished" podID="4b88874a-3d0a-4029-8581-7c76a39201a2" containerID="dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65" exitCode=60 Oct 30 13:30:02.640677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.640652 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" event={"ID":"4b88874a-3d0a-4029-8581-7c76a39201a2","Type":"ContainerDied","Data":"dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65"} Oct 30 13:30:02.640848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.640831 5049 scope.go:122] "RemoveContainer" containerID="dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65" Oct 30 13:30:02.642458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.642436 5049 generic.go:356] "Generic (PLEG): container finished" podID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerID="838283ff3745ba89feac311d7717e111cc914847fc90200d1c80ef56c75cdf88" exitCode=60 Oct 30 13:30:02.642574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.642514 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" event={"ID":"c4633f32-74bb-4ff0-ba96-7baef8fd130f","Type":"ContainerDied","Data":"838283ff3745ba89feac311d7717e111cc914847fc90200d1c80ef56c75cdf88"} Oct 30 13:30:02.642756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.642740 5049 scope.go:122] "RemoveContainer" containerID="838283ff3745ba89feac311d7717e111cc914847fc90200d1c80ef56c75cdf88" Oct 30 13:30:02.644976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.644380 5049 generic.go:356] "Generic (PLEG): container finished" podID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerID="223b88b5bdf5313447792161a0099044b0e9238539d054da71e639c68a792cdb" exitCode=60 Oct 30 13:30:02.644976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.644414 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" event={"ID":"7943ae21-45c1-466b-a9ff-9bfc8dbfe485","Type":"ContainerDied","Data":"223b88b5bdf5313447792161a0099044b0e9238539d054da71e639c68a792cdb"} Oct 30 13:30:02.644976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.644659 5049 scope.go:122] "RemoveContainer" containerID="223b88b5bdf5313447792161a0099044b0e9238539d054da71e639c68a792cdb" Oct 30 13:30:02.667350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.667282 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" podStartSLOduration=2.667248382 podStartE2EDuration="2.667248382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.665210993 +0000 UTC m=+2309.480405416" watchObservedRunningTime="2026-10-30 13:30:02.667248382 +0000 UTC m=+2309.482442806" Oct 30 13:30:02.804082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:02.804053 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:03.161134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.161094 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:03.254863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.252477 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq"] Oct 30 13:30:03.254863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.252743 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" podUID="8b67a351-2778-4edb-98ba-fb889a7150a8" containerName="maas-api" containerID="cri-o://ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9" gracePeriod=30 Oct 30 13:30:03.274597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.274528 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" podUID="8b67a351-2778-4edb-98ba-fb889a7150a8" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.206:8443/health\": dial tcp 10.132.0.206:8443: connect: connection refused" Oct 30 13:30:03.352211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.352180 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt"] Oct 30 13:30:03.368381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.368292 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:03.627573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.627537 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:30:03.660280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.660238 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" event={"ID":"8e317c46-94d2-4900-b307-6ac09a1d832d","Type":"ContainerStarted","Data":"7d43303bb7c37a54eac56f9f05534969a1e1a931cdefffc78c4df235ae73d335"} Oct 30 13:30:03.662688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.662660 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" event={"ID":"4b88874a-3d0a-4029-8581-7c76a39201a2","Type":"ContainerStarted","Data":"1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5"} Oct 30 13:30:03.662855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.662824 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" podUID="4b88874a-3d0a-4029-8581-7c76a39201a2" containerName="cleanup" containerID="cri-o://1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5" gracePeriod=30 Oct 30 13:30:03.666381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.666352 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" event={"ID":"c4633f32-74bb-4ff0-ba96-7baef8fd130f","Type":"ContainerStarted","Data":"53de5fe2314030badb126b2ef4a707c1992e0dcca5bc8c8e7d885e9b0932b098"} Oct 30 13:30:03.670230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.670159 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" event={"ID":"7943ae21-45c1-466b-a9ff-9bfc8dbfe485","Type":"ContainerStarted","Data":"3b6c56b4949fa1bd5393d3782d031664d45c74998e8d9d9e50b19222685493b7"} Oct 30 13:30:03.673749 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.673723 5049 generic.go:356] "Generic (PLEG): container finished" podID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerID="2fc0e96df1f8557f09694f45f0d14be9add21cfa66f4f7affa46bc20dd808293" exitCode=60 Oct 30 13:30:03.673864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.673807 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" event={"ID":"cc49db5d-04ab-4ca6-b0c2-9094909526b2","Type":"ContainerDied","Data":"2fc0e96df1f8557f09694f45f0d14be9add21cfa66f4f7affa46bc20dd808293"} Oct 30 13:30:03.674066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.674048 5049 scope.go:122] "RemoveContainer" containerID="2fc0e96df1f8557f09694f45f0d14be9add21cfa66f4f7affa46bc20dd808293" Oct 30 13:30:03.675482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.675458 5049 generic.go:356] "Generic (PLEG): container finished" podID="8b67a351-2778-4edb-98ba-fb889a7150a8" containerID="ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9" exitCode=0 Oct 30 13:30:03.675584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.675501 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" Oct 30 13:30:03.675650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.675579 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" event={"ID":"8b67a351-2778-4edb-98ba-fb889a7150a8","Type":"ContainerDied","Data":"ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9"} Oct 30 13:30:03.675650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.675621 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq" event={"ID":"8b67a351-2778-4edb-98ba-fb889a7150a8","Type":"ContainerDied","Data":"39c92ec375ec87b2641911ec5e7c8280c44d90c33ee90c27af2a431849d249b9"} Oct 30 13:30:03.675842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.675653 5049 scope.go:122] "RemoveContainer" containerID="ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9" Oct 30 13:30:03.678072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.678051 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" event={"ID":"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2","Type":"ContainerStarted","Data":"797c00b2df95f20d8d3217dd85d788b5bf437c505ad3d973b499e30869c3baf1"} Oct 30 13:30:03.680255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.680234 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" event={"ID":"978b344c-24b0-4fc2-a8f0-7c72d7510ceb","Type":"ContainerStarted","Data":"eb650a2fcfeb2513218b346c4fc159556e8ab833dd450464010dca312dc9f5d7"} Oct 30 13:30:03.697513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.697482 5049 scope.go:122] "RemoveContainer" containerID="ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9" Oct 30 13:30:03.698074 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:03.698045 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9\": container with ID starting with ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9 not found: ID does not exist" containerID="ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9" Oct 30 13:30:03.698208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.698089 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9"} err="failed to get container status \"ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9\": rpc error: code = NotFound desc = could not find container \"ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9\": container with ID starting with ba6c3b10f9fc11009a9122c440013fd01f5f62ef65cb21514b0a5ade7d7eecb9 not found: ID does not exist" Oct 30 13:30:03.857442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.854389 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" podStartSLOduration=3.854295671 podStartE2EDuration="3.854295671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.851710843 +0000 UTC m=+2310.666905268" watchObservedRunningTime="2026-10-30 13:30:03.854295671 +0000 UTC m=+2310.669490094" Oct 30 13:30:03.897444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.876602 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b67a351-2778-4edb-98ba-fb889a7150a8-kube-api-access-7rvls\" (UniqueName: \"kubernetes.io/projected/8b67a351-2778-4edb-98ba-fb889a7150a8-kube-api-access-7rvls\") pod \"8b67a351-2778-4edb-98ba-fb889a7150a8\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " Oct 30 13:30:03.912077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.912044 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-maas-api-tls\") pod \"8b67a351-2778-4edb-98ba-fb889a7150a8\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " Oct 30 13:30:03.912773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.912751 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-metrics-certs\") pod \"8b67a351-2778-4edb-98ba-fb889a7150a8\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " Oct 30 13:30:03.913304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.913285 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b67a351-2778-4edb-98ba-fb889a7150a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b67a351-2778-4edb-98ba-fb889a7150a8-openshift-service-ca\") pod \"8b67a351-2778-4edb-98ba-fb889a7150a8\" (UID: \"8b67a351-2778-4edb-98ba-fb889a7150a8\") " Oct 30 13:30:03.914867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.913833 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b67a351-2778-4edb-98ba-fb889a7150a8-openshift-service-ca" pod "8b67a351-2778-4edb-98ba-fb889a7150a8" (UID: "8b67a351-2778-4edb-98ba-fb889a7150a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:03.917650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.917627 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b67a351-2778-4edb-98ba-fb889a7150a8-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:03.931270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.926430 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-maas-api-tls" pod "8b67a351-2778-4edb-98ba-fb889a7150a8" (UID: "8b67a351-2778-4edb-98ba-fb889a7150a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:30:03.933843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.931550 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-metrics-certs" pod "8b67a351-2778-4edb-98ba-fb889a7150a8" (UID: "8b67a351-2778-4edb-98ba-fb889a7150a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:30:03.937574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.937549 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b67a351-2778-4edb-98ba-fb889a7150a8-kube-api-access-7rvls" pod "8b67a351-2778-4edb-98ba-fb889a7150a8" (UID: "8b67a351-2778-4edb-98ba-fb889a7150a8"). InnerVolumeSpecName "kube-api-access-7rvls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:03.939708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.939657 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" podStartSLOduration=3.9396362480000002 podStartE2EDuration="3.939636248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.933697166 +0000 UTC m=+2310.748891589" watchObservedRunningTime="2026-10-30 13:30:03.939636248 +0000 UTC m=+2310.754830653" Oct 30 13:30:03.993254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:03.993204 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" podStartSLOduration=3.9931901119999997 podStartE2EDuration="3.993190112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.987263132 +0000 UTC m=+2310.802457555" watchObservedRunningTime="2026-10-30 13:30:03.993190112 +0000 UTC m=+2310.808384534" Oct 30 13:30:04.019456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.019427 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rvls\" (UniqueName: \"kubernetes.io/projected/8b67a351-2778-4edb-98ba-fb889a7150a8-kube-api-access-7rvls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:04.019572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.019460 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:04.019572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.019474 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b67a351-2778-4edb-98ba-fb889a7150a8-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:04.025823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.025777 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" podStartSLOduration=4.025765022 podStartE2EDuration="4.025765022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.023723601 +0000 UTC m=+2310.838918024" watchObservedRunningTime="2026-10-30 13:30:04.025765022 +0000 UTC m=+2310.840959475" Oct 30 13:30:04.064957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.064906 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" podStartSLOduration=4.064893316 podStartE2EDuration="4.064893316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.061791936 +0000 UTC m=+2310.876986359" watchObservedRunningTime="2026-10-30 13:30:04.064893316 +0000 UTC m=+2310.880087738" Oct 30 13:30:04.104757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.104708 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" podStartSLOduration=4.104693653 podStartE2EDuration="4.104693653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.099530319 +0000 UTC m=+2310.914724742" watchObservedRunningTime="2026-10-30 13:30:04.104693653 +0000 UTC m=+2310.919888076" Oct 30 13:30:04.136014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.135981 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq"] Oct 30 13:30:04.141260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.141232 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-376c89-8bd64fb6d-7gdhq"] Oct 30 13:30:04.154016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.153852 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:04.266773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.266737 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946"] Oct 30 13:30:04.271289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.271261 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-376c89-hj946"] Oct 30 13:30:04.313951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.313922 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:04.506832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.506739 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:04.693543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.693507 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" event={"ID":"cc49db5d-04ab-4ca6-b0c2-9094909526b2","Type":"ContainerStarted","Data":"77cea3ace4715cf70036906bd77b4146bdee1878f2eedc6de580632e06afd4da"} Oct 30 13:30:04.698121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.698082 5049 generic.go:356] "Generic (PLEG): container finished" podID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerID="797c00b2df95f20d8d3217dd85d788b5bf437c505ad3d973b499e30869c3baf1" exitCode=60 Oct 30 13:30:04.698347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.698286 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" event={"ID":"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2","Type":"ContainerDied","Data":"797c00b2df95f20d8d3217dd85d788b5bf437c505ad3d973b499e30869c3baf1"} Oct 30 13:30:04.698428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.698386 5049 scope.go:122] "RemoveContainer" containerID="c02ebf97c4c02f1e76ae7c7cf6c6d8460c6cfd406a2228447914893731f90851" Oct 30 13:30:04.698903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.698842 5049 scope.go:122] "RemoveContainer" containerID="797c00b2df95f20d8d3217dd85d788b5bf437c505ad3d973b499e30869c3baf1" Oct 30 13:30:04.699279 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:04.699244 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk_odh-ai-gateway-infra(4b9390be-a2b9-40e5-a70e-29cde5f7c8d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" Oct 30 13:30:04.703585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.703551 5049 generic.go:356] "Generic (PLEG): container finished" podID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerID="eb650a2fcfeb2513218b346c4fc159556e8ab833dd450464010dca312dc9f5d7" exitCode=60 Oct 30 13:30:04.703691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.703636 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" event={"ID":"978b344c-24b0-4fc2-a8f0-7c72d7510ceb","Type":"ContainerDied","Data":"eb650a2fcfeb2513218b346c4fc159556e8ab833dd450464010dca312dc9f5d7"} Oct 30 13:30:04.704016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.704001 5049 scope.go:122] "RemoveContainer" containerID="eb650a2fcfeb2513218b346c4fc159556e8ab833dd450464010dca312dc9f5d7" Oct 30 13:30:04.705553 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:04.704950 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z_odh-ai-gateway-infra(978b344c-24b0-4fc2-a8f0-7c72d7510ceb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" Oct 30 13:30:04.707480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.707456 5049 generic.go:356] "Generic (PLEG): container finished" podID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerID="7d43303bb7c37a54eac56f9f05534969a1e1a931cdefffc78c4df235ae73d335" exitCode=60 Oct 30 13:30:04.707685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.707540 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" event={"ID":"8e317c46-94d2-4900-b307-6ac09a1d832d","Type":"ContainerDied","Data":"7d43303bb7c37a54eac56f9f05534969a1e1a931cdefffc78c4df235ae73d335"} Oct 30 13:30:04.708421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.707908 5049 scope.go:122] "RemoveContainer" containerID="7d43303bb7c37a54eac56f9f05534969a1e1a931cdefffc78c4df235ae73d335" Oct 30 13:30:04.708421 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:04.708310 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-t4jnj_odh-ai-gateway-infra(8e317c46-94d2-4900-b307-6ac09a1d832d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" Oct 30 13:30:04.713124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.710735 5049 generic.go:356] "Generic (PLEG): container finished" podID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerID="53de5fe2314030badb126b2ef4a707c1992e0dcca5bc8c8e7d885e9b0932b098" exitCode=60 Oct 30 13:30:04.713124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.711308 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" event={"ID":"c4633f32-74bb-4ff0-ba96-7baef8fd130f","Type":"ContainerDied","Data":"53de5fe2314030badb126b2ef4a707c1992e0dcca5bc8c8e7d885e9b0932b098"} Oct 30 13:30:04.713124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.711902 5049 scope.go:122] "RemoveContainer" containerID="53de5fe2314030badb126b2ef4a707c1992e0dcca5bc8c8e7d885e9b0932b098" Oct 30 13:30:04.713124 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:04.712316 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b_odh-ai-gateway-infra(c4633f32-74bb-4ff0-ba96-7baef8fd130f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" Oct 30 13:30:04.714689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.714667 5049 generic.go:356] "Generic (PLEG): container finished" podID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerID="3b6c56b4949fa1bd5393d3782d031664d45c74998e8d9d9e50b19222685493b7" exitCode=60 Oct 30 13:30:04.714787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.714763 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" event={"ID":"7943ae21-45c1-466b-a9ff-9bfc8dbfe485","Type":"ContainerDied","Data":"3b6c56b4949fa1bd5393d3782d031664d45c74998e8d9d9e50b19222685493b7"} Oct 30 13:30:04.715166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.715143 5049 scope.go:122] "RemoveContainer" containerID="3b6c56b4949fa1bd5393d3782d031664d45c74998e8d9d9e50b19222685493b7" Oct 30 13:30:04.715600 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:04.715578 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7_odh-ai-gateway-infra(7943ae21-45c1-466b-a9ff-9bfc8dbfe485)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" Oct 30 13:30:04.722221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.722202 5049 scope.go:122] "RemoveContainer" containerID="544094c40c0d9746195cd2c22a3ddf8f513b54a43f8b0193f03c4f402d69abc5" Oct 30 13:30:04.790926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.790890 5049 scope.go:122] "RemoveContainer" containerID="01284edf2cef1cd451e6cc38054c08637a369a95abef99439d078a803ce31989" Oct 30 13:30:04.795028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.794996 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:04.811119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.811097 5049 scope.go:122] "RemoveContainer" containerID="838283ff3745ba89feac311d7717e111cc914847fc90200d1c80ef56c75cdf88" Oct 30 13:30:04.828628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:04.828607 5049 scope.go:122] "RemoveContainer" containerID="223b88b5bdf5313447792161a0099044b0e9238539d054da71e639c68a792cdb" Oct 30 13:30:05.343604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.343572 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5993ce95-2514-401f-a815-e86f250b8d74" path="/var/lib/kubelet/pods/5993ce95-2514-401f-a815-e86f250b8d74/volumes" Oct 30 13:30:05.344049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.344033 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b67a351-2778-4edb-98ba-fb889a7150a8" path="/var/lib/kubelet/pods/8b67a351-2778-4edb-98ba-fb889a7150a8/volumes" Oct 30 13:30:05.729229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.729148 5049 scope.go:122] "RemoveContainer" containerID="53de5fe2314030badb126b2ef4a707c1992e0dcca5bc8c8e7d885e9b0932b098" Oct 30 13:30:05.729621 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:05.729594 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b_odh-ai-gateway-infra(c4633f32-74bb-4ff0-ba96-7baef8fd130f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" Oct 30 13:30:05.731154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.731132 5049 scope.go:122] "RemoveContainer" containerID="3b6c56b4949fa1bd5393d3782d031664d45c74998e8d9d9e50b19222685493b7" Oct 30 13:30:05.731534 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:05.731507 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7_odh-ai-gateway-infra(7943ae21-45c1-466b-a9ff-9bfc8dbfe485)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" Oct 30 13:30:05.732928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.732900 5049 generic.go:356] "Generic (PLEG): container finished" podID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerID="77cea3ace4715cf70036906bd77b4146bdee1878f2eedc6de580632e06afd4da" exitCode=60 Oct 30 13:30:05.733028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.732987 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" event={"ID":"cc49db5d-04ab-4ca6-b0c2-9094909526b2","Type":"ContainerDied","Data":"77cea3ace4715cf70036906bd77b4146bdee1878f2eedc6de580632e06afd4da"} Oct 30 13:30:05.733087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.733043 5049 scope.go:122] "RemoveContainer" containerID="2fc0e96df1f8557f09694f45f0d14be9add21cfa66f4f7affa46bc20dd808293" Oct 30 13:30:05.733223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.733208 5049 scope.go:122] "RemoveContainer" containerID="77cea3ace4715cf70036906bd77b4146bdee1878f2eedc6de580632e06afd4da" Oct 30 13:30:05.733602 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:05.733577 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x_odh-ai-gateway-infra(cc49db5d-04ab-4ca6-b0c2-9094909526b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" Oct 30 13:30:05.735220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.735199 5049 scope.go:122] "RemoveContainer" containerID="797c00b2df95f20d8d3217dd85d788b5bf437c505ad3d973b499e30869c3baf1" Oct 30 13:30:05.735584 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:05.735563 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk_odh-ai-gateway-infra(4b9390be-a2b9-40e5-a70e-29cde5f7c8d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" Oct 30 13:30:05.737101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.737084 5049 scope.go:122] "RemoveContainer" containerID="eb650a2fcfeb2513218b346c4fc159556e8ab833dd450464010dca312dc9f5d7" Oct 30 13:30:05.737423 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:05.737398 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z_odh-ai-gateway-infra(978b344c-24b0-4fc2-a8f0-7c72d7510ceb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" Oct 30 13:30:05.738792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:05.738774 5049 scope.go:122] "RemoveContainer" containerID="7d43303bb7c37a54eac56f9f05534969a1e1a931cdefffc78c4df235ae73d335" Oct 30 13:30:05.739025 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:05.739007 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-t4jnj_odh-ai-gateway-infra(8e317c46-94d2-4900-b307-6ac09a1d832d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" Oct 30 13:30:06.040690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.040593 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:06.236442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.236411 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:06.679367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.679303 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:06.698285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.698244 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:30:06.759140 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.759110 5049 scope.go:122] "RemoveContainer" containerID="77cea3ace4715cf70036906bd77b4146bdee1878f2eedc6de580632e06afd4da" Oct 30 13:30:06.759610 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:06.759572 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x_odh-ai-gateway-infra(cc49db5d-04ab-4ca6-b0c2-9094909526b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" Oct 30 13:30:06.762919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.762606 5049 generic.go:356] "Generic (PLEG): container finished" podID="43e04f33-e190-41e9-9c5d-2390835d027f" containerID="bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a" exitCode=0 Oct 30 13:30:06.762919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.762797 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" event={"ID":"43e04f33-e190-41e9-9c5d-2390835d027f","Type":"ContainerDied","Data":"bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a"} Oct 30 13:30:06.762919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.762863 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" Oct 30 13:30:06.762919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.762878 5049 scope.go:122] "RemoveContainer" containerID="bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a" Oct 30 13:30:06.763231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.762864 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg" event={"ID":"43e04f33-e190-41e9-9c5d-2390835d027f","Type":"ContainerDied","Data":"a4c425acba8f5430eea000060e0b7494b6fd5a47be25dd7f8ff7faecb1ce9ceb"} Oct 30 13:30:06.784035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.784005 5049 scope.go:122] "RemoveContainer" containerID="58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14" Oct 30 13:30:06.810937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.810906 5049 scope.go:122] "RemoveContainer" containerID="bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a" Oct 30 13:30:06.811340 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:06.811299 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a\": container with ID starting with bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a not found: ID does not exist" containerID="bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a" Oct 30 13:30:06.811445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.811350 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a"} err="failed to get container status \"bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a\": rpc error: code = NotFound desc = could not find container \"bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a\": container with ID starting with bc2f8205d9894e8bafe85509aa25530ac4f484479d70c5e9d8f1c4f410e2578a not found: ID does not exist" Oct 30 13:30:06.811445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.811379 5049 scope.go:122] "RemoveContainer" containerID="58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14" Oct 30 13:30:06.811731 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:06.811709 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14\": container with ID starting with 58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14 not found: ID does not exist" containerID="58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14" Oct 30 13:30:06.811803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.811738 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14"} err="failed to get container status \"58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14\": rpc error: code = NotFound desc = could not find container \"58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14\": container with ID starting with 58ded9621e33bf842360ef4c812897cef4a9f93b6e9af8a2cb9f3583d5768e14 not found: ID does not exist" Oct 30 13:30:06.866618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.866580 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-credential-socket\") pod \"43e04f33-e190-41e9-9c5d-2390835d027f\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " Oct 30 13:30:06.866811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.866737 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-certs\") pod \"43e04f33-e190-41e9-9c5d-2390835d027f\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " Oct 30 13:30:06.866884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.866823 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-kube-api-access-fb8vs\" (UniqueName: \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-kube-api-access-fb8vs\") pod \"43e04f33-e190-41e9-9c5d-2390835d027f\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " Oct 30 13:30:06.866949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.866907 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-istio-token\" (UniqueName: \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-istio-token\") pod \"43e04f33-e190-41e9-9c5d-2390835d027f\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " Oct 30 13:30:06.867014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.866963 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-socket\") pod \"43e04f33-e190-41e9-9c5d-2390835d027f\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " Oct 30 13:30:06.867014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.866983 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43e04f33-e190-41e9-9c5d-2390835d027f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e04f33-e190-41e9-9c5d-2390835d027f-istio-podinfo\") pod \"43e04f33-e190-41e9-9c5d-2390835d027f\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " Oct 30 13:30:06.867127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.866904 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-credential-socket" pod "43e04f33-e190-41e9-9c5d-2390835d027f" (UID: "43e04f33-e190-41e9-9c5d-2390835d027f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:06.867127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.867075 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-certs" pod "43e04f33-e190-41e9-9c5d-2390835d027f" (UID: "43e04f33-e190-41e9-9c5d-2390835d027f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:06.867244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.867222 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-envoy\") pod \"43e04f33-e190-41e9-9c5d-2390835d027f\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " Oct 30 13:30:06.867367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.867343 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-socket" pod "43e04f33-e190-41e9-9c5d-2390835d027f" (UID: "43e04f33-e190-41e9-9c5d-2390835d027f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:06.867503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.867478 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-data\") pod \"43e04f33-e190-41e9-9c5d-2390835d027f\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " Oct 30 13:30:06.867744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.867723 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-data" pod "43e04f33-e190-41e9-9c5d-2390835d027f" (UID: "43e04f33-e190-41e9-9c5d-2390835d027f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:06.867847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.867523 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e04f33-e190-41e9-9c5d-2390835d027f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e04f33-e190-41e9-9c5d-2390835d027f-istiod-ca-cert\") pod \"43e04f33-e190-41e9-9c5d-2390835d027f\" (UID: \"43e04f33-e190-41e9-9c5d-2390835d027f\") " Oct 30 13:30:06.867965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.867939 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e04f33-e190-41e9-9c5d-2390835d027f-istiod-ca-cert" pod "43e04f33-e190-41e9-9c5d-2390835d027f" (UID: "43e04f33-e190-41e9-9c5d-2390835d027f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:06.869491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.869466 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:06.869603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.869495 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:06.869603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.869515 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:06.869603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.869529 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:06.869603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.869543 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e04f33-e190-41e9-9c5d-2390835d027f-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:06.870171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.870145 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43e04f33-e190-41e9-9c5d-2390835d027f-istio-podinfo" pod "43e04f33-e190-41e9-9c5d-2390835d027f" (UID: "43e04f33-e190-41e9-9c5d-2390835d027f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:30:06.870265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.870240 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-istio-token" pod "43e04f33-e190-41e9-9c5d-2390835d027f" (UID: "43e04f33-e190-41e9-9c5d-2390835d027f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:06.870495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.870462 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-kube-api-access-fb8vs" pod "43e04f33-e190-41e9-9c5d-2390835d027f" (UID: "43e04f33-e190-41e9-9c5d-2390835d027f"). InnerVolumeSpecName "kube-api-access-fb8vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:06.870562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.870492 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-envoy" pod "43e04f33-e190-41e9-9c5d-2390835d027f" (UID: "43e04f33-e190-41e9-9c5d-2390835d027f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:06.971045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.970961 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb8vs\" (UniqueName: \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-kube-api-access-fb8vs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:06.971045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.970993 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e04f33-e190-41e9-9c5d-2390835d027f-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:06.971045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.971009 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e04f33-e190-41e9-9c5d-2390835d027f-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:06.971045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:06.971021 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e04f33-e190-41e9-9c5d-2390835d027f-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:07.131689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:07.131657 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:07.168217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:07.168183 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg"] Oct 30 13:30:07.194751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:07.194719 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-wnzqg"] Oct 30 13:30:07.339239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:07.339206 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" path="/var/lib/kubelet/pods/43e04f33-e190-41e9-9c5d-2390835d027f/volumes" Oct 30 13:30:07.567151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:07.567106 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:08.037156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.037124 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:08.122626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.122585 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v"] Oct 30 13:30:08.123392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.123373 5049 cpu_manager.go:414] "containerMap: removing container" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" Oct 30 13:30:08.123392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.123391 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" Oct 30 13:30:08.123527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.123431 5049 cpu_manager.go:414] "containerMap: removing container" podUID="8b67a351-2778-4edb-98ba-fb889a7150a8" containerName="maas-api" Oct 30 13:30:08.123527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.123437 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b67a351-2778-4edb-98ba-fb889a7150a8" containerName="maas-api" Oct 30 13:30:08.123676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.123659 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b67a351-2778-4edb-98ba-fb889a7150a8" containerName="maas-api" Oct 30 13:30:08.123752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.123683 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" Oct 30 13:30:08.123752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.123706 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" Oct 30 13:30:08.129369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.129348 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:30:08.157227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.157191 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-82796d-dockercfg-7rhrm\"" Oct 30 13:30:08.158493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.158462 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-82796d\"" Oct 30 13:30:08.162713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.162686 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v"] Oct 30 13:30:08.267343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.264257 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:08.292396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.290368 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4b52fc-cf26-4470-ba92-2f1d6628d313-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v\" (UID: \"7e4b52fc-cf26-4470-ba92-2f1d6628d313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:30:08.292396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.290700 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcmqn\" (UniqueName: \"kubernetes.io/projected/7e4b52fc-cf26-4470-ba92-2f1d6628d313-kube-api-access-zcmqn\") pod \"payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v\" (UID: \"7e4b52fc-cf26-4470-ba92-2f1d6628d313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:30:08.404346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.403528 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk"] Oct 30 13:30:08.418546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.418503 5049 cpu_manager.go:414] "containerMap: removing container" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" Oct 30 13:30:08.418546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.418539 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e04f33-e190-41e9-9c5d-2390835d027f" containerName="istio-proxy" Oct 30 13:30:08.432287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.432175 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcmqn\" (UniqueName: \"kubernetes.io/projected/7e4b52fc-cf26-4470-ba92-2f1d6628d313-kube-api-access-zcmqn\") pod \"payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v\" (UID: \"7e4b52fc-cf26-4470-ba92-2f1d6628d313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:30:08.432492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.432316 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4b52fc-cf26-4470-ba92-2f1d6628d313-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v\" (UID: \"7e4b52fc-cf26-4470-ba92-2f1d6628d313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:30:08.433475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.433428 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4b52fc-cf26-4470-ba92-2f1d6628d313-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v\" (UID: \"7e4b52fc-cf26-4470-ba92-2f1d6628d313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:30:08.446289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.446256 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk"] Oct 30 13:30:08.446459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.446376 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:30:08.475387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.472105 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcmqn\" (UniqueName: \"kubernetes.io/projected/7e4b52fc-cf26-4470-ba92-2f1d6628d313-kube-api-access-zcmqn\") pod \"payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v\" (UID: \"7e4b52fc-cf26-4470-ba92-2f1d6628d313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:30:08.636128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.636092 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dgvk\" (UniqueName: \"kubernetes.io/projected/b0bbdb7e-bee7-437c-b800-309359cfd827-kube-api-access-7dgvk\") pod \"payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk\" (UID: \"b0bbdb7e-bee7-437c-b800-309359cfd827\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:30:08.636392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.636296 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bbdb7e-bee7-437c-b800-309359cfd827-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk\" (UID: \"b0bbdb7e-bee7-437c-b800-309359cfd827\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:30:08.737902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.737864 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dgvk\" (UniqueName: \"kubernetes.io/projected/b0bbdb7e-bee7-437c-b800-309359cfd827-kube-api-access-7dgvk\") pod \"payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk\" (UID: \"b0bbdb7e-bee7-437c-b800-309359cfd827\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:30:08.738090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.738028 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bbdb7e-bee7-437c-b800-309359cfd827-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk\" (UID: \"b0bbdb7e-bee7-437c-b800-309359cfd827\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:30:08.739006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.738978 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bbdb7e-bee7-437c-b800-309359cfd827-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk\" (UID: \"b0bbdb7e-bee7-437c-b800-309359cfd827\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:30:08.747797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.747772 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:30:08.759742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.759715 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dgvk\" (UniqueName: \"kubernetes.io/projected/b0bbdb7e-bee7-437c-b800-309359cfd827-kube-api-access-7dgvk\") pod \"payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk\" (UID: \"b0bbdb7e-bee7-437c-b800-309359cfd827\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:30:08.763937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:08.763914 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:30:09.111795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.111767 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v"] Oct 30 13:30:09.115473 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:09.115438 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bbdb7e_bee7_437c_b800_309359cfd827.slice/crio-5f641afcc2a4ed3b55e2efad9f5ef69d7ff8d2c2481ef8e129f9de79db659f70 WatchSource:0}: Error finding container 5f641afcc2a4ed3b55e2efad9f5ef69d7ff8d2c2481ef8e129f9de79db659f70: Status 404 returned error can't find the container with id 5f641afcc2a4ed3b55e2efad9f5ef69d7ff8d2c2481ef8e129f9de79db659f70 Oct 30 13:30:09.117145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.117123 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk"] Oct 30 13:30:09.133593 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:09.133563 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4b52fc_cf26_4470_ba92_2f1d6628d313.slice/crio-086a539349cd254d9b369cb81751620e2f9b02d12cdb0ad8a177adb2f3d5f306 WatchSource:0}: Error finding container 086a539349cd254d9b369cb81751620e2f9b02d12cdb0ad8a177adb2f3d5f306: Status 404 returned error can't find the container with id 086a539349cd254d9b369cb81751620e2f9b02d12cdb0ad8a177adb2f3d5f306 Oct 30 13:30:09.374366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.374305 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:09.797247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.797207 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" event={"ID":"7e4b52fc-cf26-4470-ba92-2f1d6628d313","Type":"ContainerStarted","Data":"2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7"} Oct 30 13:30:09.797477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.797274 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" event={"ID":"7e4b52fc-cf26-4470-ba92-2f1d6628d313","Type":"ContainerStarted","Data":"086a539349cd254d9b369cb81751620e2f9b02d12cdb0ad8a177adb2f3d5f306"} Oct 30 13:30:09.797477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.797375 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:30:09.799076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.799052 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" event={"ID":"b0bbdb7e-bee7-437c-b800-309359cfd827","Type":"ContainerStarted","Data":"bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388"} Oct 30 13:30:09.799210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.799094 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" event={"ID":"b0bbdb7e-bee7-437c-b800-309359cfd827","Type":"ContainerStarted","Data":"5f641afcc2a4ed3b55e2efad9f5ef69d7ff8d2c2481ef8e129f9de79db659f70"} Oct 30 13:30:09.799210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.799133 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:30:09.847433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.847369 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" podStartSLOduration=1.847351997 podStartE2EDuration="1.847351997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:09.837646806 +0000 UTC m=+2316.652841230" watchObservedRunningTime="2026-10-30 13:30:09.847351997 +0000 UTC m=+2316.662546420" Oct 30 13:30:09.896564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.896506 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" podStartSLOduration=1.896489377 podStartE2EDuration="1.896489377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:09.891943326 +0000 UTC m=+2316.707137749" watchObservedRunningTime="2026-10-30 13:30:09.896489377 +0000 UTC m=+2316.711683800" Oct 30 13:30:09.899547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.899521 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:09.914401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.914375 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559"] Oct 30 13:30:09.920917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.920869 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:30:09.937955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.937930 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-fdd4d4\"" Oct 30 13:30:09.944243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.944218 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559"] Oct 30 13:30:09.959362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.958396 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d4c8d1-20dd-4057-9766-faca798308b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559\" (UID: \"47d4c8d1-20dd-4057-9766-faca798308b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:30:09.959362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:09.958523 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzmzc\" (UniqueName: \"kubernetes.io/projected/47d4c8d1-20dd-4057-9766-faca798308b8-kube-api-access-fzmzc\") pod \"payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559\" (UID: \"47d4c8d1-20dd-4057-9766-faca798308b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:30:10.060361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.060236 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d4c8d1-20dd-4057-9766-faca798308b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559\" (UID: \"47d4c8d1-20dd-4057-9766-faca798308b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:30:10.060529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.060372 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzmzc\" (UniqueName: \"kubernetes.io/projected/47d4c8d1-20dd-4057-9766-faca798308b8-kube-api-access-fzmzc\") pod \"payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559\" (UID: \"47d4c8d1-20dd-4057-9766-faca798308b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:30:10.069479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.062062 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d4c8d1-20dd-4057-9766-faca798308b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559\" (UID: \"47d4c8d1-20dd-4057-9766-faca798308b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:30:10.094083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.093967 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:10.133211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.132604 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzmzc\" (UniqueName: \"kubernetes.io/projected/47d4c8d1-20dd-4057-9766-faca798308b8-kube-api-access-fzmzc\") pod \"payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559\" (UID: \"47d4c8d1-20dd-4057-9766-faca798308b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:30:10.259152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.259114 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:30:10.585484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.585457 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559"] Oct 30 13:30:10.587802 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:10.587774 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d4c8d1_20dd_4057_9766_faca798308b8.slice/crio-297d608f06e8666b87d7016e65bcb63f131ae817bc38d9c9787983efec712777 WatchSource:0}: Error finding container 297d608f06e8666b87d7016e65bcb63f131ae817bc38d9c9787983efec712777: Status 404 returned error can't find the container with id 297d608f06e8666b87d7016e65bcb63f131ae817bc38d9c9787983efec712777 Oct 30 13:30:10.813193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.813151 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" event={"ID":"47d4c8d1-20dd-4057-9766-faca798308b8","Type":"ContainerStarted","Data":"fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857"} Oct 30 13:30:10.813430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.813220 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" event={"ID":"47d4c8d1-20dd-4057-9766-faca798308b8","Type":"ContainerStarted","Data":"297d608f06e8666b87d7016e65bcb63f131ae817bc38d9c9787983efec712777"} Oct 30 13:30:10.813430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.813252 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:30:10.853005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:10.852911 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" podStartSLOduration=1.852896082 podStartE2EDuration="1.852896082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:10.851515048 +0000 UTC m=+2317.666709473" watchObservedRunningTime="2026-10-30 13:30:10.852896082 +0000 UTC m=+2317.668090517" Oct 30 13:30:11.357184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:11.357123 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:11.989003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:11.988967 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:12.237553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:12.237515 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:12.330343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:12.328913 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8"] Oct 30 13:30:12.330343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:12.329155 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" containerID="cri-o://626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0" gracePeriod=30 Oct 30 13:30:12.337106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:12.337074 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc"] Oct 30 13:30:12.337346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:12.337302 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerName="istio-proxy" containerID="cri-o://aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c" gracePeriod=30 Oct 30 13:30:12.464355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:12.461930 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:12.500391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:12.495640 5049 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:30:12.500391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:12.495701 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:30:13.498280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:13.498246 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:15.328840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:15.328809 5049 scope.go:122] "RemoveContainer" containerID="797c00b2df95f20d8d3217dd85d788b5bf437c505ad3d973b499e30869c3baf1" Oct 30 13:30:15.378005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:15.375771 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:15.872166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:15.872126 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" event={"ID":"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2","Type":"ContainerStarted","Data":"9123548d69405280478400ba9d7e13169e8a50656cb578f71ef3f1038153b86c"} Oct 30 13:30:16.329067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:16.329002 5049 scope.go:122] "RemoveContainer" containerID="77cea3ace4715cf70036906bd77b4146bdee1878f2eedc6de580632e06afd4da" Oct 30 13:30:16.329764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:16.329741 5049 scope.go:122] "RemoveContainer" containerID="53de5fe2314030badb126b2ef4a707c1992e0dcca5bc8c8e7d885e9b0932b098" Oct 30 13:30:16.886018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:16.885929 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" event={"ID":"c4633f32-74bb-4ff0-ba96-7baef8fd130f","Type":"ContainerStarted","Data":"2b736354223f151b996ee8e5331837c92e7fee581dc31e824bf9bf6b49ebfb91"} Oct 30 13:30:16.888250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:16.888219 5049 generic.go:356] "Generic (PLEG): container finished" podID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerID="9123548d69405280478400ba9d7e13169e8a50656cb578f71ef3f1038153b86c" exitCode=60 Oct 30 13:30:16.888385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:16.888246 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" event={"ID":"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2","Type":"ContainerDied","Data":"9123548d69405280478400ba9d7e13169e8a50656cb578f71ef3f1038153b86c"} Oct 30 13:30:16.888385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:16.888304 5049 scope.go:122] "RemoveContainer" containerID="797c00b2df95f20d8d3217dd85d788b5bf437c505ad3d973b499e30869c3baf1" Oct 30 13:30:16.888740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:16.888721 5049 scope.go:122] "RemoveContainer" containerID="9123548d69405280478400ba9d7e13169e8a50656cb578f71ef3f1038153b86c" Oct 30 13:30:16.889125 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:16.889104 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk_odh-ai-gateway-infra(4b9390be-a2b9-40e5-a70e-29cde5f7c8d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" Oct 30 13:30:16.941292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:16.941257 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk"] Oct 30 13:30:17.127106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.127042 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" podUID="5a1441a6-7293-4524-a05f-9c0d334944d4" containerName="main" containerID="cri-o://f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1" gracePeriod=45 Oct 30 13:30:17.212009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.211972 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:17.329265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.329233 5049 scope.go:122] "RemoveContainer" containerID="3b6c56b4949fa1bd5393d3782d031664d45c74998e8d9d9e50b19222685493b7" Oct 30 13:30:17.330341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.330284 5049 scope.go:122] "RemoveContainer" containerID="7d43303bb7c37a54eac56f9f05534969a1e1a931cdefffc78c4df235ae73d335" Oct 30 13:30:17.699357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.699261 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:30:17.718893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.715533 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:30:17.722589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.722569 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:30:17.833385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833355 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-credential-socket\") pod \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " Oct 30 13:30:17.833537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833431 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1441a6-7293-4524-a05f-9c0d334944d4-kube-api-access-hddgf\" (UniqueName: \"kubernetes.io/projected/5a1441a6-7293-4524-a05f-9c0d334944d4-kube-api-access-hddgf\") pod \"5a1441a6-7293-4524-a05f-9c0d334944d4\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " Oct 30 13:30:17.833537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833461 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-credential-socket\") pod \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " Oct 30 13:30:17.833537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833506 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-podinfo\") pod \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " Oct 30 13:30:17.833655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833551 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-token\" (UniqueName: \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-token\") pod \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " Oct 30 13:30:17.833655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833583 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istiod-ca-cert\") pod \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " Oct 30 13:30:17.833655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833646 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-data\") pod \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " Oct 30 13:30:17.833754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833681 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-kube-api-access-4qwqh\" (UniqueName: \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-kube-api-access-4qwqh\") pod \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " Oct 30 13:30:17.833754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833685 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-credential-socket" pod "e54e3b09-d1da-44b3-9f72-0371d05e26c4" (UID: "e54e3b09-d1da-44b3-9f72-0371d05e26c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.833825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833792 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istiod-ca-cert\") pod \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " Oct 30 13:30:17.833861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833824 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-model-cache\") pod \"5a1441a6-7293-4524-a05f-9c0d334944d4\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " Oct 30 13:30:17.833924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833857 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-kube-api-access-gfpcv\" (UniqueName: \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-kube-api-access-gfpcv\") pod \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " Oct 30 13:30:17.833924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833881 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-certs\") pod \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " Oct 30 13:30:17.833924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833905 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-tmp-dir\") pod \"5a1441a6-7293-4524-a05f-9c0d334944d4\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " Oct 30 13:30:17.833924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833880 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-credential-socket" pod "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" (UID: "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.833924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833923 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-data\") pod \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " Oct 30 13:30:17.834189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833960 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-home\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-home\") pod \"5a1441a6-7293-4524-a05f-9c0d334944d4\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " Oct 30 13:30:17.834189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.833987 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-socket\") pod \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " Oct 30 13:30:17.834189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.834007 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-certs\") pod \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " Oct 30 13:30:17.834189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.834030 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-socket\") pod \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " Oct 30 13:30:17.834189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.834044 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istiod-ca-cert" pod "e54e3b09-d1da-44b3-9f72-0371d05e26c4" (UID: "e54e3b09-d1da-44b3-9f72-0371d05e26c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:17.834189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.834073 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-envoy\") pod \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " Oct 30 13:30:17.834189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.834101 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a1441a6-7293-4524-a05f-9c0d334944d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a1441a6-7293-4524-a05f-9c0d334944d4-tls-certs\") pod \"5a1441a6-7293-4524-a05f-9c0d334944d4\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " Oct 30 13:30:17.834189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.834126 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-dshm\") pod \"5a1441a6-7293-4524-a05f-9c0d334944d4\" (UID: \"5a1441a6-7293-4524-a05f-9c0d334944d4\") " Oct 30 13:30:17.834189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.834174 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-envoy\") pod \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " Oct 30 13:30:17.834695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.834281 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-certs" pod "e54e3b09-d1da-44b3-9f72-0371d05e26c4" (UID: "e54e3b09-d1da-44b3-9f72-0371d05e26c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.834695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.834367 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-token\" (UniqueName: \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-token\") pod \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\" (UID: \"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7\") " Oct 30 13:30:17.834695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.834412 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-podinfo\") pod \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\" (UID: \"e54e3b09-d1da-44b3-9f72-0371d05e26c4\") " Oct 30 13:30:17.835527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.835266 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-tmp-dir" pod "5a1441a6-7293-4524-a05f-9c0d334944d4" (UID: "5a1441a6-7293-4524-a05f-9c0d334944d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.835527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.835288 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istiod-ca-cert" pod "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" (UID: "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:17.835527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.835314 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-model-cache" pod "5a1441a6-7293-4524-a05f-9c0d334944d4" (UID: "5a1441a6-7293-4524-a05f-9c0d334944d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.835527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.835476 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-data" pod "e54e3b09-d1da-44b3-9f72-0371d05e26c4" (UID: "e54e3b09-d1da-44b3-9f72-0371d05e26c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.835852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.835776 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-socket" pod "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" (UID: "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.836291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.835931 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-home" pod "5a1441a6-7293-4524-a05f-9c0d334944d4" (UID: "5a1441a6-7293-4524-a05f-9c0d334944d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.836291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.835976 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-certs" pod "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" (UID: "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.836291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.835992 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-data" pod "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" (UID: "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.836291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.836005 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-socket" pod "e54e3b09-d1da-44b3-9f72-0371d05e26c4" (UID: "e54e3b09-d1da-44b3-9f72-0371d05e26c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.836747 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-token" pod "e54e3b09-d1da-44b3-9f72-0371d05e26c4" (UID: "e54e3b09-d1da-44b3-9f72-0371d05e26c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837036 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837055 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837069 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837083 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837096 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837110 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837122 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837134 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837147 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837161 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837173 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837186 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837200 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.837240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.837214 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.838277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.838241 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-podinfo" pod "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" (UID: "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:30:17.839500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.839476 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-envoy" pod "e54e3b09-d1da-44b3-9f72-0371d05e26c4" (UID: "e54e3b09-d1da-44b3-9f72-0371d05e26c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.839965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.839941 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-envoy" pod "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" (UID: "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.840124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.840103 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-kube-api-access-gfpcv" pod "e54e3b09-d1da-44b3-9f72-0371d05e26c4" (UID: "e54e3b09-d1da-44b3-9f72-0371d05e26c4"). InnerVolumeSpecName "kube-api-access-gfpcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.840189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.840138 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1441a6-7293-4524-a05f-9c0d334944d4-kube-api-access-hddgf" pod "5a1441a6-7293-4524-a05f-9c0d334944d4" (UID: "5a1441a6-7293-4524-a05f-9c0d334944d4"). InnerVolumeSpecName "kube-api-access-hddgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.840485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.840462 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-token" pod "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" (UID: "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.840990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.840957 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-dshm" pod "5a1441a6-7293-4524-a05f-9c0d334944d4" (UID: "5a1441a6-7293-4524-a05f-9c0d334944d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.842107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.842089 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-podinfo" pod "e54e3b09-d1da-44b3-9f72-0371d05e26c4" (UID: "e54e3b09-d1da-44b3-9f72-0371d05e26c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:30:17.842450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.842429 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-kube-api-access-4qwqh" pod "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" (UID: "3e60a821-b3b0-4653-b7e9-8ab6cb2818a7"). InnerVolumeSpecName "kube-api-access-4qwqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.842538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.842513 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a1441a6-7293-4524-a05f-9c0d334944d4-tls-certs" pod "5a1441a6-7293-4524-a05f-9c0d334944d4" (UID: "5a1441a6-7293-4524-a05f-9c0d334944d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:30:17.901607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.901569 5049 generic.go:356] "Generic (PLEG): container finished" podID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerID="2b736354223f151b996ee8e5331837c92e7fee581dc31e824bf9bf6b49ebfb91" exitCode=60 Oct 30 13:30:17.901770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.901635 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" event={"ID":"c4633f32-74bb-4ff0-ba96-7baef8fd130f","Type":"ContainerDied","Data":"2b736354223f151b996ee8e5331837c92e7fee581dc31e824bf9bf6b49ebfb91"} Oct 30 13:30:17.901770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.901693 5049 scope.go:122] "RemoveContainer" containerID="53de5fe2314030badb126b2ef4a707c1992e0dcca5bc8c8e7d885e9b0932b098" Oct 30 13:30:17.902144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.902122 5049 scope.go:122] "RemoveContainer" containerID="2b736354223f151b996ee8e5331837c92e7fee581dc31e824bf9bf6b49ebfb91" Oct 30 13:30:17.902630 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:17.902602 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b_odh-ai-gateway-infra(c4633f32-74bb-4ff0-ba96-7baef8fd130f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.905799 5049 generic.go:356] "Generic (PLEG): container finished" podID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerID="aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c" exitCode=0 Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.905985 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" event={"ID":"e54e3b09-d1da-44b3-9f72-0371d05e26c4","Type":"ContainerDied","Data":"aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c"} Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.906033 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" event={"ID":"e54e3b09-d1da-44b3-9f72-0371d05e26c4","Type":"ContainerDied","Data":"930c864d92adeb18d7164523b9b3c0cb0d23f81dae560956a3851bf8150b62d0"} Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.906111 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc" Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.917114 5049 generic.go:356] "Generic (PLEG): container finished" podID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerID="0c48121989aa7de6974243995571f0ee0160d369b6fd7b57f1f916dd4f31ac1f" exitCode=60 Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.917232 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" event={"ID":"cc49db5d-04ab-4ca6-b0c2-9094909526b2","Type":"ContainerDied","Data":"0c48121989aa7de6974243995571f0ee0160d369b6fd7b57f1f916dd4f31ac1f"} Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.917690 5049 scope.go:122] "RemoveContainer" containerID="0c48121989aa7de6974243995571f0ee0160d369b6fd7b57f1f916dd4f31ac1f" Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:17.918704 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x_odh-ai-gateway-infra(cc49db5d-04ab-4ca6-b0c2-9094909526b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.921647 5049 generic.go:356] "Generic (PLEG): container finished" podID="5a1441a6-7293-4524-a05f-9c0d334944d4" containerID="f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1" exitCode=0 Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.921686 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.921712 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" event={"ID":"5a1441a6-7293-4524-a05f-9c0d334944d4","Type":"ContainerDied","Data":"f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1"} Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.921742 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k" event={"ID":"5a1441a6-7293-4524-a05f-9c0d334944d4","Type":"ContainerDied","Data":"445011d0d36738d2ebd7e3766fbcacac17e51d84a039c1e351fb86f50ac9d3d2"} Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.925946 5049 generic.go:356] "Generic (PLEG): container finished" podID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerID="626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0" exitCode=0 Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.926020 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" event={"ID":"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7","Type":"ContainerDied","Data":"626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0"} Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.926118 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" event={"ID":"3e60a821-b3b0-4653-b7e9-8ab6cb2818a7","Type":"ContainerDied","Data":"8b581ae5a91d5492e6cbceb6389a501bc6ffdf6e6c2bd0317d388ec62c01323f"} Oct 30 13:30:17.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.926256 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8" Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.940625 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b"] Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.941465 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfpcv\" (UniqueName: \"kubernetes.io/projected/e54e3b09-d1da-44b3-9f72-0371d05e26c4-kube-api-access-gfpcv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.941489 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.941504 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a1441a6-7293-4524-a05f-9c0d334944d4-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.941516 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a1441a6-7293-4524-a05f-9c0d334944d4-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.941704 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.942301 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.942345 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54e3b09-d1da-44b3-9f72-0371d05e26c4-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.942361 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hddgf\" (UniqueName: \"kubernetes.io/projected/5a1441a6-7293-4524-a05f-9c0d334944d4-kube-api-access-hddgf\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.942375 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.949742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.942394 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qwqh\" (UniqueName: \"kubernetes.io/projected/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7-kube-api-access-4qwqh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.961796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.961424 5049 scope.go:122] "RemoveContainer" containerID="aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c" Oct 30 13:30:17.979630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.979592 5049 scope.go:122] "RemoveContainer" containerID="aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c" Oct 30 13:30:17.980051 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:17.980025 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c\": container with ID starting with aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c not found: ID does not exist" containerID="aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c" Oct 30 13:30:17.980117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.980061 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c"} err="failed to get container status \"aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c\": rpc error: code = NotFound desc = could not find container \"aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c\": container with ID starting with aede90f316458d48470190f60b4217cc94ec39238042b4f022812c3380da6b6c not found: ID does not exist" Oct 30 13:30:17.980117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.980090 5049 scope.go:122] "RemoveContainer" containerID="77cea3ace4715cf70036906bd77b4146bdee1878f2eedc6de580632e06afd4da" Oct 30 13:30:17.996445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:17.996370 5049 scope.go:122] "RemoveContainer" containerID="f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1" Oct 30 13:30:18.009639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.009617 5049 scope.go:122] "RemoveContainer" containerID="f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1" Oct 30 13:30:18.010279 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:18.010142 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1\": container with ID starting with f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1 not found: ID does not exist" containerID="f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1" Oct 30 13:30:18.010279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.010179 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1"} err="failed to get container status \"f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1\": rpc error: code = NotFound desc = could not find container \"f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1\": container with ID starting with f8716fad6175427336ac68a90de98c024562988ed14d6272ca2e729e8199a2c1 not found: ID does not exist" Oct 30 13:30:18.010279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.010207 5049 scope.go:122] "RemoveContainer" containerID="626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0" Oct 30 13:30:18.023105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.023087 5049 scope.go:122] "RemoveContainer" containerID="626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0" Oct 30 13:30:18.023436 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:18.023379 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0\": container with ID starting with 626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0 not found: ID does not exist" containerID="626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0" Oct 30 13:30:18.023507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.023444 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0"} err="failed to get container status \"626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0\": rpc error: code = NotFound desc = could not find container \"626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0\": container with ID starting with 626347ba7442cd18a3f49d252d42362837f46423f4c274900efb910553f785b0 not found: ID does not exist" Oct 30 13:30:18.126919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.126889 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" Oct 30 13:30:18.148087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.148054 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-maas-api-service-ca\") pod \"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2\" (UID: \"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2\") " Oct 30 13:30:18.148206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.148146 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-kube-api-access-fksvv\" (UniqueName: \"kubernetes.io/projected/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-kube-api-access-fksvv\") pod \"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2\" (UID: \"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2\") " Oct 30 13:30:18.148710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.148685 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-maas-api-service-ca" pod "4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" (UID: "4b9390be-a2b9-40e5-a70e-29cde5f7c8d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:18.151157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.151120 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-kube-api-access-fksvv" pod "4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" (UID: "4b9390be-a2b9-40e5-a70e-29cde5f7c8d2"). InnerVolumeSpecName "kube-api-access-fksvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:18.154272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.154204 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k"] Oct 30 13:30:18.168480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.168458 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/auth-test-model-9c6033-kserve-b868bcd77-sbw8k"] Oct 30 13:30:18.214636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.213036 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc"] Oct 30 13:30:18.219743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.219711 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-pmrpc"] Oct 30 13:30:18.250155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.250124 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fksvv\" (UniqueName: \"kubernetes.io/projected/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-kube-api-access-fksvv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.250155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.250151 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.261144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.261118 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8"] Oct 30 13:30:18.265468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.265447 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-ljjs8"] Oct 30 13:30:18.329258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.329226 5049 scope.go:122] "RemoveContainer" containerID="eb650a2fcfeb2513218b346c4fc159556e8ab833dd450464010dca312dc9f5d7" Oct 30 13:30:18.941678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.941172 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" Oct 30 13:30:18.941678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.941169 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk" event={"ID":"4b9390be-a2b9-40e5-a70e-29cde5f7c8d2","Type":"ContainerDied","Data":"b99ebe8d45f36bacff4f071a028f6ad94d6195c3080d0bcdb78ad9ff4b04981d"} Oct 30 13:30:18.941678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.941396 5049 scope.go:122] "RemoveContainer" containerID="9123548d69405280478400ba9d7e13169e8a50656cb578f71ef3f1038153b86c" Oct 30 13:30:18.945691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.945580 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" event={"ID":"978b344c-24b0-4fc2-a8f0-7c72d7510ceb","Type":"ContainerStarted","Data":"279ea66f87eb6d21a2226e308526a44d4ccc1dfb24de64e911bba46da714421a"} Oct 30 13:30:18.949361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.948039 5049 generic.go:356] "Generic (PLEG): container finished" podID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerID="f44a792793b37ba4f112ccceab343b59d29c93a942a31b8bc929652d794de5ae" exitCode=60 Oct 30 13:30:18.949361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.948112 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" event={"ID":"8e317c46-94d2-4900-b307-6ac09a1d832d","Type":"ContainerDied","Data":"f44a792793b37ba4f112ccceab343b59d29c93a942a31b8bc929652d794de5ae"} Oct 30 13:30:18.949361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.948634 5049 scope.go:122] "RemoveContainer" containerID="f44a792793b37ba4f112ccceab343b59d29c93a942a31b8bc929652d794de5ae" Oct 30 13:30:18.949361 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:18.948967 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-t4jnj_odh-ai-gateway-infra(8e317c46-94d2-4900-b307-6ac09a1d832d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" Oct 30 13:30:18.953700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.953679 5049 generic.go:356] "Generic (PLEG): container finished" podID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerID="73d7ec5e21ae89cc202bcf829f5d095755dadfa8e4d00dc8a81fc808d744553f" exitCode=60 Oct 30 13:30:18.953800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.953751 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" event={"ID":"7943ae21-45c1-466b-a9ff-9bfc8dbfe485","Type":"ContainerDied","Data":"73d7ec5e21ae89cc202bcf829f5d095755dadfa8e4d00dc8a81fc808d744553f"} Oct 30 13:30:18.954566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.954164 5049 scope.go:122] "RemoveContainer" containerID="73d7ec5e21ae89cc202bcf829f5d095755dadfa8e4d00dc8a81fc808d744553f" Oct 30 13:30:18.954566 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:18.954505 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7_odh-ai-gateway-infra(7943ae21-45c1-466b-a9ff-9bfc8dbfe485)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" Oct 30 13:30:18.956722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.956704 5049 scope.go:122] "RemoveContainer" containerID="7d43303bb7c37a54eac56f9f05534969a1e1a931cdefffc78c4df235ae73d335" Oct 30 13:30:18.971542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:18.971276 5049 scope.go:122] "RemoveContainer" containerID="3b6c56b4949fa1bd5393d3782d031664d45c74998e8d9d9e50b19222685493b7" Oct 30 13:30:19.041911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.039383 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x"] Oct 30 13:30:19.077380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.070208 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk"] Oct 30 13:30:19.077380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.075988 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889450-zm8xk"] Oct 30 13:30:19.269113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.265522 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:19.277049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.277021 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" Oct 30 13:30:19.340908 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.340872 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" path="/var/lib/kubelet/pods/3e60a821-b3b0-4653-b7e9-8ab6cb2818a7/volumes" Oct 30 13:30:19.341884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.341858 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" path="/var/lib/kubelet/pods/4b9390be-a2b9-40e5-a70e-29cde5f7c8d2/volumes" Oct 30 13:30:19.342730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.342710 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a1441a6-7293-4524-a05f-9c0d334944d4" path="/var/lib/kubelet/pods/5a1441a6-7293-4524-a05f-9c0d334944d4/volumes" Oct 30 13:30:19.343959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.343937 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" path="/var/lib/kubelet/pods/e54e3b09-d1da-44b3-9f72-0371d05e26c4/volumes" Oct 30 13:30:19.367787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.367765 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4633f32-74bb-4ff0-ba96-7baef8fd130f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4633f32-74bb-4ff0-ba96-7baef8fd130f-maas-api-service-ca\") pod \"c4633f32-74bb-4ff0-ba96-7baef8fd130f\" (UID: \"c4633f32-74bb-4ff0-ba96-7baef8fd130f\") " Oct 30 13:30:19.368070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.368056 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4633f32-74bb-4ff0-ba96-7baef8fd130f-kube-api-access-88cxz\" (UniqueName: \"kubernetes.io/projected/c4633f32-74bb-4ff0-ba96-7baef8fd130f-kube-api-access-88cxz\") pod \"c4633f32-74bb-4ff0-ba96-7baef8fd130f\" (UID: \"c4633f32-74bb-4ff0-ba96-7baef8fd130f\") " Oct 30 13:30:19.368131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.368101 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4633f32-74bb-4ff0-ba96-7baef8fd130f-maas-api-service-ca" pod "c4633f32-74bb-4ff0-ba96-7baef8fd130f" (UID: "c4633f32-74bb-4ff0-ba96-7baef8fd130f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:19.368586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.368567 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4633f32-74bb-4ff0-ba96-7baef8fd130f-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.370502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.370313 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4633f32-74bb-4ff0-ba96-7baef8fd130f-kube-api-access-88cxz" pod "c4633f32-74bb-4ff0-ba96-7baef8fd130f" (UID: "c4633f32-74bb-4ff0-ba96-7baef8fd130f"). InnerVolumeSpecName "kube-api-access-88cxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:19.373103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.373082 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" Oct 30 13:30:19.469771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.469736 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc49db5d-04ab-4ca6-b0c2-9094909526b2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc49db5d-04ab-4ca6-b0c2-9094909526b2-maas-api-service-ca\") pod \"cc49db5d-04ab-4ca6-b0c2-9094909526b2\" (UID: \"cc49db5d-04ab-4ca6-b0c2-9094909526b2\") " Oct 30 13:30:19.469948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.469837 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc49db5d-04ab-4ca6-b0c2-9094909526b2-kube-api-access-k8dh5\" (UniqueName: \"kubernetes.io/projected/cc49db5d-04ab-4ca6-b0c2-9094909526b2-kube-api-access-k8dh5\") pod \"cc49db5d-04ab-4ca6-b0c2-9094909526b2\" (UID: \"cc49db5d-04ab-4ca6-b0c2-9094909526b2\") " Oct 30 13:30:19.470173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.470146 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc49db5d-04ab-4ca6-b0c2-9094909526b2-maas-api-service-ca" pod "cc49db5d-04ab-4ca6-b0c2-9094909526b2" (UID: "cc49db5d-04ab-4ca6-b0c2-9094909526b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:19.470681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.470655 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc49db5d-04ab-4ca6-b0c2-9094909526b2-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.470813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.470684 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88cxz\" (UniqueName: \"kubernetes.io/projected/c4633f32-74bb-4ff0-ba96-7baef8fd130f-kube-api-access-88cxz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.472523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.472499 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc49db5d-04ab-4ca6-b0c2-9094909526b2-kube-api-access-k8dh5" pod "cc49db5d-04ab-4ca6-b0c2-9094909526b2" (UID: "cc49db5d-04ab-4ca6-b0c2-9094909526b2"). InnerVolumeSpecName "kube-api-access-k8dh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:19.571618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.571587 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8dh5\" (UniqueName: \"kubernetes.io/projected/cc49db5d-04ab-4ca6-b0c2-9094909526b2-kube-api-access-k8dh5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.967998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.967911 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" Oct 30 13:30:19.967998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.967943 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b" event={"ID":"c4633f32-74bb-4ff0-ba96-7baef8fd130f","Type":"ContainerDied","Data":"7f650d5b1fde7665dbfdc1c7fd5041ef1745858ad4e4484cf08815c5a43079b4"} Oct 30 13:30:19.968503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.968030 5049 scope.go:122] "RemoveContainer" containerID="2b736354223f151b996ee8e5331837c92e7fee581dc31e824bf9bf6b49ebfb91" Oct 30 13:30:19.971553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.971529 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" Oct 30 13:30:19.971698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.971550 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x" event={"ID":"cc49db5d-04ab-4ca6-b0c2-9094909526b2","Type":"ContainerDied","Data":"7ff8ed8e117a70d0ae12d563ab023c3216c599528f240d15cdee0918e2d70e14"} Oct 30 13:30:19.975187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.975159 5049 generic.go:356] "Generic (PLEG): container finished" podID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerID="279ea66f87eb6d21a2226e308526a44d4ccc1dfb24de64e911bba46da714421a" exitCode=60 Oct 30 13:30:19.975306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.975235 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" event={"ID":"978b344c-24b0-4fc2-a8f0-7c72d7510ceb","Type":"ContainerDied","Data":"279ea66f87eb6d21a2226e308526a44d4ccc1dfb24de64e911bba46da714421a"} Oct 30 13:30:19.975621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.975604 5049 scope.go:122] "RemoveContainer" containerID="279ea66f87eb6d21a2226e308526a44d4ccc1dfb24de64e911bba46da714421a" Oct 30 13:30:19.975907 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:19.975886 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z_odh-ai-gateway-infra(978b344c-24b0-4fc2-a8f0-7c72d7510ceb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" Oct 30 13:30:19.981900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.981878 5049 scope.go:122] "RemoveContainer" containerID="0c48121989aa7de6974243995571f0ee0160d369b6fd7b57f1f916dd4f31ac1f" Oct 30 13:30:19.991763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:19.991746 5049 scope.go:122] "RemoveContainer" containerID="eb650a2fcfeb2513218b346c4fc159556e8ab833dd450464010dca312dc9f5d7" Oct 30 13:30:20.012519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.012490 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z"] Oct 30 13:30:20.051014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.050984 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj"] Oct 30 13:30:20.079470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.079435 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b"] Oct 30 13:30:20.085953 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.085793 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889450-f6f8b"] Oct 30 13:30:20.162309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.162274 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x"] Oct 30 13:30:20.166420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.166393 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889450-cgl2x"] Oct 30 13:30:20.173680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.173368 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7"] Oct 30 13:30:20.260334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.260284 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" Oct 30 13:30:20.283023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.282996 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e317c46-94d2-4900-b307-6ac09a1d832d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e317c46-94d2-4900-b307-6ac09a1d832d-maas-api-service-ca\") pod \"8e317c46-94d2-4900-b307-6ac09a1d832d\" (UID: \"8e317c46-94d2-4900-b307-6ac09a1d832d\") " Oct 30 13:30:20.283161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.283062 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e317c46-94d2-4900-b307-6ac09a1d832d-kube-api-access-4fh7g\" (UniqueName: \"kubernetes.io/projected/8e317c46-94d2-4900-b307-6ac09a1d832d-kube-api-access-4fh7g\") pod \"8e317c46-94d2-4900-b307-6ac09a1d832d\" (UID: \"8e317c46-94d2-4900-b307-6ac09a1d832d\") " Oct 30 13:30:20.283416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.283392 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e317c46-94d2-4900-b307-6ac09a1d832d-maas-api-service-ca" pod "8e317c46-94d2-4900-b307-6ac09a1d832d" (UID: "8e317c46-94d2-4900-b307-6ac09a1d832d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:20.283730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.283712 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e317c46-94d2-4900-b307-6ac09a1d832d-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.286027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.286004 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e317c46-94d2-4900-b307-6ac09a1d832d-kube-api-access-4fh7g" pod "8e317c46-94d2-4900-b307-6ac09a1d832d" (UID: "8e317c46-94d2-4900-b307-6ac09a1d832d"). InnerVolumeSpecName "kube-api-access-4fh7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:20.343884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.343852 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" Oct 30 13:30:20.399811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.399774 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-maas-api-service-ca\") pod \"7943ae21-45c1-466b-a9ff-9bfc8dbfe485\" (UID: \"7943ae21-45c1-466b-a9ff-9bfc8dbfe485\") " Oct 30 13:30:20.400014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.399902 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-kube-api-access-b8g7m\" (UniqueName: \"kubernetes.io/projected/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-kube-api-access-b8g7m\") pod \"7943ae21-45c1-466b-a9ff-9bfc8dbfe485\" (UID: \"7943ae21-45c1-466b-a9ff-9bfc8dbfe485\") " Oct 30 13:30:20.401995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.401963 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fh7g\" (UniqueName: \"kubernetes.io/projected/8e317c46-94d2-4900-b307-6ac09a1d832d-kube-api-access-4fh7g\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.406395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.406355 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-kube-api-access-b8g7m" pod "7943ae21-45c1-466b-a9ff-9bfc8dbfe485" (UID: "7943ae21-45c1-466b-a9ff-9bfc8dbfe485"). InnerVolumeSpecName "kube-api-access-b8g7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:20.406552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.406534 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-maas-api-service-ca" pod "7943ae21-45c1-466b-a9ff-9bfc8dbfe485" (UID: "7943ae21-45c1-466b-a9ff-9bfc8dbfe485"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:20.503221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.503128 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.503221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.503167 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8g7m\" (UniqueName: \"kubernetes.io/projected/7943ae21-45c1-466b-a9ff-9bfc8dbfe485-kube-api-access-b8g7m\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.817385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.817353 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:30:20.817581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.817540 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:30:20.990475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.990440 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" event={"ID":"7943ae21-45c1-466b-a9ff-9bfc8dbfe485","Type":"ContainerDied","Data":"e3e265ed8d27c935a8991603eb759623adad2819f6bb5e21600e68a47945b824"} Oct 30 13:30:20.990835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.990506 5049 scope.go:122] "RemoveContainer" containerID="73d7ec5e21ae89cc202bcf829f5d095755dadfa8e4d00dc8a81fc808d744553f" Oct 30 13:30:20.990835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:20.990626 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7" Oct 30 13:30:21.005090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.005058 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" event={"ID":"8e317c46-94d2-4900-b307-6ac09a1d832d","Type":"ContainerDied","Data":"4e5660dce5ccb80ec386d4fb19d3c5e039d68eb7788d11ff85e293f026e568cb"} Oct 30 13:30:21.005202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.005104 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj" Oct 30 13:30:21.032677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.031571 5049 scope.go:122] "RemoveContainer" containerID="f44a792793b37ba4f112ccceab343b59d29c93a942a31b8bc929652d794de5ae" Oct 30 13:30:21.048957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.048918 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7"] Oct 30 13:30:21.054404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.054372 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889450-hczp7"] Oct 30 13:30:21.133564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.132899 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj"] Oct 30 13:30:21.138461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.138429 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-t4jnj"] Oct 30 13:30:21.215244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.215206 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" Oct 30 13:30:21.233553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.233517 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:21.314796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.314754 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-maas-api-service-ca\") pod \"978b344c-24b0-4fc2-a8f0-7c72d7510ceb\" (UID: \"978b344c-24b0-4fc2-a8f0-7c72d7510ceb\") " Oct 30 13:30:21.315519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.315399 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-maas-api-service-ca" pod "978b344c-24b0-4fc2-a8f0-7c72d7510ceb" (UID: "978b344c-24b0-4fc2-a8f0-7c72d7510ceb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:21.315761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.315742 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-kube-api-access-c8fb6\" (UniqueName: \"kubernetes.io/projected/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-kube-api-access-c8fb6\") pod \"978b344c-24b0-4fc2-a8f0-7c72d7510ceb\" (UID: \"978b344c-24b0-4fc2-a8f0-7c72d7510ceb\") " Oct 30 13:30:21.317005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.316985 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.326020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.325986 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-kube-api-access-c8fb6" pod "978b344c-24b0-4fc2-a8f0-7c72d7510ceb" (UID: "978b344c-24b0-4fc2-a8f0-7c72d7510ceb"). InnerVolumeSpecName "kube-api-access-c8fb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:21.338296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.338267 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" path="/var/lib/kubelet/pods/7943ae21-45c1-466b-a9ff-9bfc8dbfe485/volumes" Oct 30 13:30:21.338797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.338778 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" path="/var/lib/kubelet/pods/8e317c46-94d2-4900-b307-6ac09a1d832d/volumes" Oct 30 13:30:21.339237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.339218 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" path="/var/lib/kubelet/pods/c4633f32-74bb-4ff0-ba96-7baef8fd130f/volumes" Oct 30 13:30:21.340016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.339996 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" path="/var/lib/kubelet/pods/cc49db5d-04ab-4ca6-b0c2-9094909526b2/volumes" Oct 30 13:30:21.419183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.419147 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8fb6\" (UniqueName: \"kubernetes.io/projected/978b344c-24b0-4fc2-a8f0-7c72d7510ceb-kube-api-access-c8fb6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.829461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:21.829430 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:30:22.018871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:22.018841 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" Oct 30 13:30:22.019355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:22.018884 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z" event={"ID":"978b344c-24b0-4fc2-a8f0-7c72d7510ceb","Type":"ContainerDied","Data":"c3491c833adba7da4e80ba44ec722d646f5534c3b00b33f56b12ae3bac2f80c6"} Oct 30 13:30:22.019355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:22.018950 5049 scope.go:122] "RemoveContainer" containerID="279ea66f87eb6d21a2226e308526a44d4ccc1dfb24de64e911bba46da714421a" Oct 30 13:30:22.067988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:22.067956 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z"] Oct 30 13:30:22.074016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:22.073986 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889450-wmr7z"] Oct 30 13:30:23.114440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:23.114385 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:23.340068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:23.340028 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" path="/var/lib/kubelet/pods/978b344c-24b0-4fc2-a8f0-7c72d7510ceb/volumes" Oct 30 13:30:25.365660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:25.365623 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:26.678282 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:26.678246 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:30:27.431270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:27.431232 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:28.012203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.012166 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8743-8d775f"] Oct 30 13:30:28.012958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.012938 5049 cpu_manager.go:414] "containerMap: removing container" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerName="cleanup" Oct 30 13:30:28.012958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.012957 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerName="cleanup" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.012969 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5a1441a6-7293-4524-a05f-9c0d334944d4" containerName="main" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.012975 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1441a6-7293-4524-a05f-9c0d334944d4" containerName="main" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.012989 5049 cpu_manager.go:414] "containerMap: removing container" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerName="cleanup" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.012995 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerName="cleanup" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013001 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013018 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013029 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerName="cleanup" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013034 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerName="cleanup" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013041 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerName="cleanup" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013046 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerName="cleanup" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013075 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerName="cleanup" Oct 30 13:30:28.013081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013080 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013089 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013094 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013103 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerName="istio-proxy" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013108 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerName="istio-proxy" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013133 5049 cpu_manager.go:414] "containerMap: removing container" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013138 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013149 5049 cpu_manager.go:414] "containerMap: removing container" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013153 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013162 5049 cpu_manager.go:414] "containerMap: removing container" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013166 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013172 5049 cpu_manager.go:414] "containerMap: removing container" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013177 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013184 5049 cpu_manager.go:414] "containerMap: removing container" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013189 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013196 5049 cpu_manager.go:414] "containerMap: removing container" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013200 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013207 5049 cpu_manager.go:414] "containerMap: removing container" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013212 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013224 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013228 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013235 5049 cpu_manager.go:414] "containerMap: removing container" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013239 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013248 5049 cpu_manager.go:414] "containerMap: removing container" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013256 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013376 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013384 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013392 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013398 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013404 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013411 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1441a6-7293-4524-a05f-9c0d334944d4" containerName="main" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013419 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e317c46-94d2-4900-b307-6ac09a1d832d" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013425 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e60a821-b3b0-4653-b7e9-8ab6cb2818a7" containerName="istio-proxy" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013433 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013440 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013447 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013452 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4633f32-74bb-4ff0-ba96-7baef8fd130f" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013457 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54e3b09-d1da-44b3-9f72-0371d05e26c4" containerName="istio-proxy" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013466 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013473 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013479 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013484 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013488 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013493 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="7943ae21-45c1-466b-a9ff-9bfc8dbfe485" containerName="cleanup" Oct 30 13:30:28.013724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.013500 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="978b344c-24b0-4fc2-a8f0-7c72d7510ceb" containerName="cleanup" Oct 30 13:30:28.018752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.018730 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:28.038550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.038509 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8743-8d775f"] Oct 30 13:30:28.041124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.039099 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nqkjx\"" Oct 30 13:30:28.068773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.068741 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlm42\" (UniqueName: \"kubernetes.io/projected/cee0245a-154f-418e-8c3d-bef96af5b57a-kube-api-access-mlm42\") pod \"test-curl-8743-8d775f\" (UID: \"cee0245a-154f-418e-8c3d-bef96af5b57a\") " pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:28.169663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.169631 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlm42\" (UniqueName: \"kubernetes.io/projected/cee0245a-154f-418e-8c3d-bef96af5b57a-kube-api-access-mlm42\") pod \"test-curl-8743-8d775f\" (UID: \"cee0245a-154f-418e-8c3d-bef96af5b57a\") " pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:28.191122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.191092 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlm42\" (UniqueName: \"kubernetes.io/projected/cee0245a-154f-418e-8c3d-bef96af5b57a-kube-api-access-mlm42\") pod \"test-curl-8743-8d775f\" (UID: \"cee0245a-154f-418e-8c3d-bef96af5b57a\") " pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:28.346265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.346231 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:28.660762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.660676 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:28.672447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:28.672293 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8743-8d775f"] Oct 30 13:30:28.675751 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:28.675709 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcee0245a_154f_418e_8c3d_bef96af5b57a.slice/crio-65eb17808733e2ea17e5ed144deeb0ec0fd06094ac91da6aae0c611555864bcc WatchSource:0}: Error finding container 65eb17808733e2ea17e5ed144deeb0ec0fd06094ac91da6aae0c611555864bcc: Status 404 returned error can't find the container with id 65eb17808733e2ea17e5ed144deeb0ec0fd06094ac91da6aae0c611555864bcc Oct 30 13:30:29.128667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:29.128633 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8743-8d775f" event={"ID":"cee0245a-154f-418e-8c3d-bef96af5b57a","Type":"ContainerStarted","Data":"65eb17808733e2ea17e5ed144deeb0ec0fd06094ac91da6aae0c611555864bcc"} Oct 30 13:30:29.596858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:29.596822 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:30.289620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:30.289569 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:31.152051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:31.151956 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8743-8d775f" event={"ID":"cee0245a-154f-418e-8c3d-bef96af5b57a","Type":"ContainerStarted","Data":"12af5847ae77ac4e39fd0fdeff75624cc63728260b90ac9700164da916df9a31"} Oct 30 13:30:31.188170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:31.188120 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8743-8d775f" podStartSLOduration=2.053352932 podStartE2EDuration="4.18810557s" totalImagesPullingTime="2.134752638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:31.187453376 +0000 UTC m=+2338.002647799" watchObservedRunningTime="2026-10-30 13:30:31.18810557 +0000 UTC m=+2338.003299992" Oct 30 13:30:31.468001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:31.467914 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:32.121075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.120247 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8743-8d775f"] Oct 30 13:30:32.126956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.126923 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8743-8d775f"] Oct 30 13:30:32.289872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.289833 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8743-41e576"] Oct 30 13:30:32.290642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.290624 5049 cpu_manager.go:414] "containerMap: removing container" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" containerName="test-curl-8743-8d775f" Oct 30 13:30:32.290701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.290644 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" containerName="test-curl-8743-8d775f" Oct 30 13:30:32.290701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.290693 5049 cpu_manager.go:414] "containerMap: removing container" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerName="cleanup" Oct 30 13:30:32.290701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.290699 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc49db5d-04ab-4ca6-b0c2-9094909526b2" containerName="cleanup" Oct 30 13:30:32.290802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.290719 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerName="cleanup" Oct 30 13:30:32.290802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.290724 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerName="cleanup" Oct 30 13:30:32.290870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.290842 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9390be-a2b9-40e5-a70e-29cde5f7c8d2" containerName="cleanup" Oct 30 13:30:32.290870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.290855 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" containerName="test-curl-8743-8d775f" Oct 30 13:30:32.296063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.296036 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:32.317885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.314829 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8743-41e576"] Oct 30 13:30:32.359783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.359744 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xjcc\" (UniqueName: \"kubernetes.io/projected/d631a5c7-c279-4da8-9385-762802ecc58b-kube-api-access-8xjcc\") pod \"test-curl-8743-41e576\" (UID: \"d631a5c7-c279-4da8-9385-762802ecc58b\") " pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:32.460756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.460655 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xjcc\" (UniqueName: \"kubernetes.io/projected/d631a5c7-c279-4da8-9385-762802ecc58b-kube-api-access-8xjcc\") pod \"test-curl-8743-41e576\" (UID: \"d631a5c7-c279-4da8-9385-762802ecc58b\") " pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:32.491466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.491438 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xjcc\" (UniqueName: \"kubernetes.io/projected/d631a5c7-c279-4da8-9385-762802ecc58b-kube-api-access-8xjcc\") pod \"test-curl-8743-41e576\" (UID: \"d631a5c7-c279-4da8-9385-762802ecc58b\") " pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:32.613333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.613293 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:32.930347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:32.928502 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8743-41e576"] Oct 30 13:30:32.932845 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:32.932811 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd631a5c7_c279_4da8_9385_762802ecc58b.slice/crio-448bfe275fbbb2dfda31008e9f2cd12867aa85f3b7662192aa4dd04ea9cf99e8 WatchSource:0}: Error finding container 448bfe275fbbb2dfda31008e9f2cd12867aa85f3b7662192aa4dd04ea9cf99e8: Status 404 returned error can't find the container with id 448bfe275fbbb2dfda31008e9f2cd12867aa85f3b7662192aa4dd04ea9cf99e8 Oct 30 13:30:33.177483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:33.177448 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8743-41e576" event={"ID":"d631a5c7-c279-4da8-9385-762802ecc58b","Type":"ContainerStarted","Data":"bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f"} Oct 30 13:30:33.177483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:33.177493 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8743-41e576" event={"ID":"d631a5c7-c279-4da8-9385-762802ecc58b","Type":"ContainerStarted","Data":"448bfe275fbbb2dfda31008e9f2cd12867aa85f3b7662192aa4dd04ea9cf99e8"} Oct 30 13:30:33.177770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:33.177621 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8743-8d775f" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" containerName="test-curl-8743-8d775f" containerID="cri-o://12af5847ae77ac4e39fd0fdeff75624cc63728260b90ac9700164da916df9a31" gracePeriod=2 Oct 30 13:30:33.216166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:33.216060 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8743-41e576" podStartSLOduration=1.21604515 podStartE2EDuration="1.21604515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:33.214222141 +0000 UTC m=+2340.029416565" watchObservedRunningTime="2026-10-30 13:30:33.21604515 +0000 UTC m=+2340.031239570" Oct 30 13:30:33.225790 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:33.225757 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-8d775f\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:33.237414 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:33.237379 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-8d775f\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:33.317415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:33.317379 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j"] Oct 30 13:30:33.370915 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:33.370880 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-8d775f\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:33.397160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:33.397128 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:33.555789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:33.555759 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:33.962504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:33.962477 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" Oct 30 13:30:34.085891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.085797 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b88874a-3d0a-4029-8581-7c76a39201a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b88874a-3d0a-4029-8581-7c76a39201a2-maas-api-service-ca\") pod \"4b88874a-3d0a-4029-8581-7c76a39201a2\" (UID: \"4b88874a-3d0a-4029-8581-7c76a39201a2\") " Oct 30 13:30:34.086095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.086053 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b88874a-3d0a-4029-8581-7c76a39201a2-kube-api-access-nxvdq\" (UniqueName: \"kubernetes.io/projected/4b88874a-3d0a-4029-8581-7c76a39201a2-kube-api-access-nxvdq\") pod \"4b88874a-3d0a-4029-8581-7c76a39201a2\" (UID: \"4b88874a-3d0a-4029-8581-7c76a39201a2\") " Oct 30 13:30:34.086208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.086186 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b88874a-3d0a-4029-8581-7c76a39201a2-maas-api-service-ca" pod "4b88874a-3d0a-4029-8581-7c76a39201a2" (UID: "4b88874a-3d0a-4029-8581-7c76a39201a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:34.086682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.086661 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b88874a-3d0a-4029-8581-7c76a39201a2-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:34.088566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.088542 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b88874a-3d0a-4029-8581-7c76a39201a2-kube-api-access-nxvdq" pod "4b88874a-3d0a-4029-8581-7c76a39201a2" (UID: "4b88874a-3d0a-4029-8581-7c76a39201a2"). InnerVolumeSpecName "kube-api-access-nxvdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:34.187926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.187892 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxvdq\" (UniqueName: \"kubernetes.io/projected/4b88874a-3d0a-4029-8581-7c76a39201a2-kube-api-access-nxvdq\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:34.189159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.189133 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8743-41e576"] Oct 30 13:30:34.189411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.189381 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" Oct 30 13:30:34.189411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.189384 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" event={"ID":"4b88874a-3d0a-4029-8581-7c76a39201a2","Type":"ContainerDied","Data":"1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5"} Oct 30 13:30:34.189593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.189291 5049 generic.go:356] "Generic (PLEG): container finished" podID="4b88874a-3d0a-4029-8581-7c76a39201a2" containerID="1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5" exitCode=137 Oct 30 13:30:34.189593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.189440 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt" event={"ID":"4b88874a-3d0a-4029-8581-7c76a39201a2","Type":"ContainerDied","Data":"4613ad0658821d33b845f709e491a27315973d5c3456200cbb4e279bc58b43d2"} Oct 30 13:30:34.189593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.189469 5049 scope.go:122] "RemoveContainer" containerID="1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5" Oct 30 13:30:34.195435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.195415 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8743-41e576"] Oct 30 13:30:34.203997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.203963 5049 scope.go:122] "RemoveContainer" containerID="dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65" Oct 30 13:30:34.224539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.223755 5049 scope.go:122] "RemoveContainer" containerID="1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5" Oct 30 13:30:34.224539 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:34.224371 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5\": container with ID starting with 1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5 not found: ID does not exist" containerID="1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5" Oct 30 13:30:34.224539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.224410 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5"} err="failed to get container status \"1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5\": rpc error: code = NotFound desc = could not find container \"1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5\": container with ID starting with 1b3d9f2669caf026d236c26db1ffe74821eae4cfa89229d3bbec6f4814da24c5 not found: ID does not exist" Oct 30 13:30:34.224539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.224434 5049 scope.go:122] "RemoveContainer" containerID="dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65" Oct 30 13:30:34.224862 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:34.224757 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65\": container with ID starting with dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65 not found: ID does not exist" containerID="dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65" Oct 30 13:30:34.224862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.224795 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65"} err="failed to get container status \"dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65\": rpc error: code = NotFound desc = could not find container \"dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65\": container with ID starting with dc42959186e98bccb16e1fe8e10230c30de33b19d6bd2e4280d364cb4fdb7b65 not found: ID does not exist" Oct 30 13:30:34.258391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.253333 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt"] Oct 30 13:30:34.258391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.257410 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-376c89-29889450-dc9wt"] Oct 30 13:30:34.793672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.793629 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk"] Oct 30 13:30:34.794390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.794372 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4b88874a-3d0a-4029-8581-7c76a39201a2" containerName="cleanup" Oct 30 13:30:34.794390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.794389 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b88874a-3d0a-4029-8581-7c76a39201a2" containerName="cleanup" Oct 30 13:30:34.794507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.794407 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4b88874a-3d0a-4029-8581-7c76a39201a2" containerName="cleanup" Oct 30 13:30:34.794507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.794412 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b88874a-3d0a-4029-8581-7c76a39201a2" containerName="cleanup" Oct 30 13:30:34.794507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.794438 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d631a5c7-c279-4da8-9385-762802ecc58b" containerName="test-curl-8743-41e576" Oct 30 13:30:34.794507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.794443 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d631a5c7-c279-4da8-9385-762802ecc58b" containerName="test-curl-8743-41e576" Oct 30 13:30:34.794717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.794545 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b88874a-3d0a-4029-8581-7c76a39201a2" containerName="cleanup" Oct 30 13:30:34.794717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.794556 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b88874a-3d0a-4029-8581-7c76a39201a2" containerName="cleanup" Oct 30 13:30:34.794717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.794566 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d631a5c7-c279-4da8-9385-762802ecc58b" containerName="test-curl-8743-41e576" Oct 30 13:30:34.799651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.799630 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:34.821466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.821438 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:30:34.822158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.821852 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:30:34.830954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.830926 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk"] Oct 30 13:30:34.898092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.898063 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brkr\" (UniqueName: \"kubernetes.io/projected/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-kube-api-access-9brkr\") pod \"maas-api-revoke-sub-275ab4bf-vnzdk\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:34.898238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.898129 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-275ab4bf-vnzdk\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:34.898238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.898198 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-275ab4bf-vnzdk\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:35.000006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:34.999956 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brkr\" (UniqueName: \"kubernetes.io/projected/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-kube-api-access-9brkr\") pod \"maas-api-revoke-sub-275ab4bf-vnzdk\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:35.000189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.000065 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-275ab4bf-vnzdk\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:35.000189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.000108 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-275ab4bf-vnzdk\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:35.000839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.000813 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-275ab4bf-vnzdk\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:35.002586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.002568 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-275ab4bf-vnzdk\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:35.022983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.022955 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brkr\" (UniqueName: \"kubernetes.io/projected/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-kube-api-access-9brkr\") pod \"maas-api-revoke-sub-275ab4bf-vnzdk\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:35.114641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.114561 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:35.207794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.207759 5049 generic.go:356] "Generic (PLEG): container finished" podID="cee0245a-154f-418e-8c3d-bef96af5b57a" containerID="12af5847ae77ac4e39fd0fdeff75624cc63728260b90ac9700164da916df9a31" exitCode=137 Oct 30 13:30:35.209542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.209516 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8743-41e576" podUID="d631a5c7-c279-4da8-9385-762802ecc58b" containerName="test-curl-8743-41e576" containerID="cri-o://bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f" gracePeriod=2 Oct 30 13:30:35.223542 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:35.223505 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-41e576\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="d631a5c7-c279-4da8-9385-762802ecc58b" pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:35.348371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.348126 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b88874a-3d0a-4029-8581-7c76a39201a2" path="/var/lib/kubelet/pods/4b88874a-3d0a-4029-8581-7c76a39201a2/volumes" Oct 30 13:30:35.401899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.401872 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:35.413629 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:35.413595 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-8d775f\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:35.429303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.429282 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk"] Oct 30 13:30:35.432381 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:35.432351 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47006709_da8c_4ef7_9eaf_e8de0a6ecdda.slice/crio-9543e139bf7db4c868e5eed2733d6d9c64de2736e022132ca29ba11f2d1453f4 WatchSource:0}: Error finding container 9543e139bf7db4c868e5eed2733d6d9c64de2736e022132ca29ba11f2d1453f4: Status 404 returned error can't find the container with id 9543e139bf7db4c868e5eed2733d6d9c64de2736e022132ca29ba11f2d1453f4 Oct 30 13:30:35.507954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.507928 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee0245a-154f-418e-8c3d-bef96af5b57a-kube-api-access-mlm42\" (UniqueName: \"kubernetes.io/projected/cee0245a-154f-418e-8c3d-bef96af5b57a-kube-api-access-mlm42\") pod \"cee0245a-154f-418e-8c3d-bef96af5b57a\" (UID: \"cee0245a-154f-418e-8c3d-bef96af5b57a\") " Oct 30 13:30:35.510717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.510691 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee0245a-154f-418e-8c3d-bef96af5b57a-kube-api-access-mlm42" pod "cee0245a-154f-418e-8c3d-bef96af5b57a" (UID: "cee0245a-154f-418e-8c3d-bef96af5b57a"). InnerVolumeSpecName "kube-api-access-mlm42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:35.610638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.610603 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlm42\" (UniqueName: \"kubernetes.io/projected/cee0245a-154f-418e-8c3d-bef96af5b57a-kube-api-access-mlm42\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:35.676231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.676161 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp"] Oct 30 13:30:35.680931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.680909 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.693756 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:35.693029 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-8d775f\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:35.694178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.694160 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f0a31b\"/\"kube-root-ca.crt\"" Oct 30 13:30:35.695347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.694856 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f0a31b\"/\"default-dockercfg-vgg4c\"" Oct 30 13:30:35.697033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.697015 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f0a31b\"/\"openshift-service-ca.crt\"" Oct 30 13:30:35.705963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.705939 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp"] Oct 30 13:30:35.791540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.791513 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:35.795245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.795227 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f0a31b\"/\"auth-test-model-f0a31b-kserve-self-signed-certs\"" Oct 30 13:30:35.815620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.815593 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-tmp-dir\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.815713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.815664 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-dshm\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.815818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.815790 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-model-cache\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.815917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.815896 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2mdz\" (UniqueName: \"kubernetes.io/projected/936e185a-1af8-4104-a8fb-ab6c2c291e60-kube-api-access-b2mdz\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.816012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.815997 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-home\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.816163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.816145 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936e185a-1af8-4104-a8fb-ab6c2c291e60-tls-certs\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.903768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.903739 5049 scope.go:122] "RemoveContainer" containerID="f46a367166175e3dfb700b594ccb38f663b414d43389eccb1f5c8e342931c2c5" Oct 30 13:30:35.916911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.916891 5049 scope.go:122] "RemoveContainer" containerID="32543271073ea644a851e7ffd79e41d2adcfadda8b1f44da2d6404bc0e684116" Oct 30 13:30:35.917263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.917240 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-model-cache\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.917399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.917332 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2mdz\" (UniqueName: \"kubernetes.io/projected/936e185a-1af8-4104-a8fb-ab6c2c291e60-kube-api-access-b2mdz\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.917467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.917441 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-home\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.917607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.917574 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936e185a-1af8-4104-a8fb-ab6c2c291e60-tls-certs\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.917728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.917647 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-model-cache\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.917728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.917682 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-tmp-dir\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.917858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.917738 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-dshm\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.917946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.917921 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-home\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.918134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.918112 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-tmp-dir\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.920536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.920511 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-dshm\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.920724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.920708 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936e185a-1af8-4104-a8fb-ab6c2c291e60-tls-certs\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.966342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.958418 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2mdz\" (UniqueName: \"kubernetes.io/projected/936e185a-1af8-4104-a8fb-ab6c2c291e60-kube-api-access-b2mdz\") pod \"auth-test-model-f0a31b-kserve-d78d4774f-bvkfp\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:35.979603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:35.978760 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:36.020498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:36.020204 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:36.046432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:36.046399 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w"] Oct 30 13:30:36.056160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:36.056136 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167051bd-vc57w"] Oct 30 13:30:36.231821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:36.231735 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:36.231821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:36.231783 5049 scope.go:122] "RemoveContainer" containerID="12af5847ae77ac4e39fd0fdeff75624cc63728260b90ac9700164da916df9a31" Oct 30 13:30:36.236122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:36.236091 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" event={"ID":"47006709-da8c-4ef7-9eaf-e8de0a6ecdda","Type":"ContainerStarted","Data":"b0d9326272248f3ec62cbf9ef886e85656120d4c0d6636b0e1d519a34e2874c8"} Oct 30 13:30:36.236552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:36.236522 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" event={"ID":"47006709-da8c-4ef7-9eaf-e8de0a6ecdda","Type":"ContainerStarted","Data":"9543e139bf7db4c868e5eed2733d6d9c64de2736e022132ca29ba11f2d1453f4"} Oct 30 13:30:36.243958 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:36.243922 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-8d775f\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:36.288646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:36.288602 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" podStartSLOduration=2.288587839 podStartE2EDuration="2.288587839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:36.284537262 +0000 UTC m=+2343.099731686" watchObservedRunningTime="2026-10-30 13:30:36.288587839 +0000 UTC m=+2343.103782253" Oct 30 13:30:36.295942 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:36.295912 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-8d775f\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:36.310365 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:36.308461 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-8d775f\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:36.375796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:36.375770 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp"] Oct 30 13:30:36.378790 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:36.378762 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod936e185a_1af8_4104_a8fb_ab6c2c291e60.slice/crio-a070eedac25c167f27115c75b9dfd61d3256fcfc8069bd112566d16d4b579172 WatchSource:0}: Error finding container a070eedac25c167f27115c75b9dfd61d3256fcfc8069bd112566d16d4b579172: Status 404 returned error can't find the container with id a070eedac25c167f27115c75b9dfd61d3256fcfc8069bd112566d16d4b579172 Oct 30 13:30:37.248840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.248805 5049 generic.go:356] "Generic (PLEG): container finished" podID="47006709-da8c-4ef7-9eaf-e8de0a6ecdda" containerID="b0d9326272248f3ec62cbf9ef886e85656120d4c0d6636b0e1d519a34e2874c8" exitCode=0 Oct 30 13:30:37.249389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.248914 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" event={"ID":"47006709-da8c-4ef7-9eaf-e8de0a6ecdda","Type":"ContainerDied","Data":"b0d9326272248f3ec62cbf9ef886e85656120d4c0d6636b0e1d519a34e2874c8"} Oct 30 13:30:37.251392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.251368 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" event={"ID":"936e185a-1af8-4104-a8fb-ab6c2c291e60","Type":"ContainerStarted","Data":"429d936f990e6bfde7480e58927b7ae5fd690af40c85382a796276bd9deb6460"} Oct 30 13:30:37.251566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.251406 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" event={"ID":"936e185a-1af8-4104-a8fb-ab6c2c291e60","Type":"ContainerStarted","Data":"a070eedac25c167f27115c75b9dfd61d3256fcfc8069bd112566d16d4b579172"} Oct 30 13:30:37.260706 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:37.260670 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-8d775f\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:37.342964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.342917 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a1f076-8806-4777-bf33-bb4460d9735e" path="/var/lib/kubelet/pods/52a1f076-8806-4777-bf33-bb4460d9735e/volumes" Oct 30 13:30:37.343539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.343512 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" path="/var/lib/kubelet/pods/cee0245a-154f-418e-8c3d-bef96af5b57a/volumes" Oct 30 13:30:37.344635 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:37.344605 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-8d775f\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="cee0245a-154f-418e-8c3d-bef96af5b57a" pod="opendatahub/test-curl-8743-8d775f" Oct 30 13:30:37.397912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.397841 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" podStartSLOduration=2.397820749 podStartE2EDuration="2.397820749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:37.394140467 +0000 UTC m=+2344.209334887" watchObservedRunningTime="2026-10-30 13:30:37.397820749 +0000 UTC m=+2344.213015173" Oct 30 13:30:37.459790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.459762 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:37.547810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.547774 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d631a5c7-c279-4da8-9385-762802ecc58b-kube-api-access-8xjcc\" (UniqueName: \"kubernetes.io/projected/d631a5c7-c279-4da8-9385-762802ecc58b-kube-api-access-8xjcc\") pod \"d631a5c7-c279-4da8-9385-762802ecc58b\" (UID: \"d631a5c7-c279-4da8-9385-762802ecc58b\") " Oct 30 13:30:37.550901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.550544 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d631a5c7-c279-4da8-9385-762802ecc58b-kube-api-access-8xjcc" pod "d631a5c7-c279-4da8-9385-762802ecc58b" (UID: "d631a5c7-c279-4da8-9385-762802ecc58b"). InnerVolumeSpecName "kube-api-access-8xjcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:37.650835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.650795 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xjcc\" (UniqueName: \"kubernetes.io/projected/d631a5c7-c279-4da8-9385-762802ecc58b-kube-api-access-8xjcc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:37.835725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:37.835621 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:38.022120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.022074 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:38.263839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.263753 5049 generic.go:356] "Generic (PLEG): container finished" podID="d631a5c7-c279-4da8-9385-762802ecc58b" containerID="bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f" exitCode=137 Oct 30 13:30:38.263839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.263802 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:38.263839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.263828 5049 scope.go:122] "RemoveContainer" containerID="bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f" Oct 30 13:30:38.277150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.277125 5049 scope.go:122] "RemoveContainer" containerID="bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f" Oct 30 13:30:38.277474 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:38.277451 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f\": container with ID starting with bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f not found: ID does not exist" containerID="bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f" Oct 30 13:30:38.277555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.277481 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f"} err="failed to get container status \"bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f\": rpc error: code = NotFound desc = could not find container \"bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f\": container with ID starting with bac4c7733425fbeeb66fec875b88f7e51b6681c8650f10b4bb97566b8dea2e0f not found: ID does not exist" Oct 30 13:30:38.285058 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:38.285033 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-41e576\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="d631a5c7-c279-4da8-9385-762802ecc58b" pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:38.448752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.448722 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:38.496947 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:38.496907 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-41e576\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="d631a5c7-c279-4da8-9385-762802ecc58b" pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:38.568356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.565382 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-xks-ca\") pod \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " Oct 30 13:30:38.568356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.565516 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-kube-api-access-9brkr\" (UniqueName: \"kubernetes.io/projected/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-kube-api-access-9brkr\") pod \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " Oct 30 13:30:38.568356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.565631 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-openshift-ca\") pod \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\" (UID: \"47006709-da8c-4ef7-9eaf-e8de0a6ecdda\") " Oct 30 13:30:38.568356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.566951 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-openshift-ca" pod "47006709-da8c-4ef7-9eaf-e8de0a6ecdda" (UID: "47006709-da8c-4ef7-9eaf-e8de0a6ecdda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:38.577936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.569916 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-xks-ca" pod "47006709-da8c-4ef7-9eaf-e8de0a6ecdda" (UID: "47006709-da8c-4ef7-9eaf-e8de0a6ecdda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:30:38.579757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.579724 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-kube-api-access-9brkr" pod "47006709-da8c-4ef7-9eaf-e8de0a6ecdda" (UID: "47006709-da8c-4ef7-9eaf-e8de0a6ecdda"). InnerVolumeSpecName "kube-api-access-9brkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:38.668452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.668416 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:38.668452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.668450 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:38.668452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:38.668460 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brkr\" (UniqueName: \"kubernetes.io/projected/47006709-da8c-4ef7-9eaf-e8de0a6ecdda-kube-api-access-9brkr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:39.275197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:39.275163 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" Oct 30 13:30:39.275197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:39.275190 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk" event={"ID":"47006709-da8c-4ef7-9eaf-e8de0a6ecdda","Type":"ContainerDied","Data":"9543e139bf7db4c868e5eed2733d6d9c64de2736e022132ca29ba11f2d1453f4"} Oct 30 13:30:39.275769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:39.275272 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9543e139bf7db4c868e5eed2733d6d9c64de2736e022132ca29ba11f2d1453f4" Oct 30 13:30:39.309494 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:39.309455 5049 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-41e576\" is forbidden: User \"system:node:ip-10-0-133-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-61.ec2.internal' and this object" podUID="d631a5c7-c279-4da8-9385-762802ecc58b" pod="opendatahub/test-curl-8743-41e576" Oct 30 13:30:39.339900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:39.339872 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d631a5c7-c279-4da8-9385-762802ecc58b" path="/var/lib/kubelet/pods/d631a5c7-c279-4da8-9385-762802ecc58b/volumes" Oct 30 13:30:39.690086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:39.690051 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:41.590817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:41.590782 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:41.971054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:41.970972 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:42.132179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:42.132133 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:43.349094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.349064 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:43.529308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.529270 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q"] Oct 30 13:30:43.530210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.530185 5049 cpu_manager.go:414] "containerMap: removing container" podUID="47006709-da8c-4ef7-9eaf-e8de0a6ecdda" containerName="revoke-keys" Oct 30 13:30:43.530358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.530211 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47006709-da8c-4ef7-9eaf-e8de0a6ecdda" containerName="revoke-keys" Oct 30 13:30:43.530490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.530475 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="47006709-da8c-4ef7-9eaf-e8de0a6ecdda" containerName="revoke-keys" Oct 30 13:30:43.535975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.535948 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.550178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.550152 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-cda0d9-openshift-default-dockercfg-jlswf\"" Oct 30 13:30:43.568834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.568805 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q"] Oct 30 13:30:43.640828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.640738 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-podinfo\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.640828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.640773 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd91c2b7-2153-434b-8a1c-817eabac78b0-istiod-ca-cert\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.640828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.640798 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-data\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.641097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.640988 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.641097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.641045 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-certs\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.641165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.641152 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-credential-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.641200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.641180 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9bw\" (UniqueName: \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-kube-api-access-dw9bw\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.641254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.641202 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-token\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.641369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.641349 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-envoy\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.742655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.742620 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9bw\" (UniqueName: \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-kube-api-access-dw9bw\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.742810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.742664 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-token\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.742810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.742725 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-envoy\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.742943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.742839 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-podinfo\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.742943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.742867 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd91c2b7-2153-434b-8a1c-817eabac78b0-istiod-ca-cert\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.743121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.743080 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-data\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.743444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.743417 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-data\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.743646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.743438 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.743774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.743707 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-certs\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.743774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.743713 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.743892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.743788 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd91c2b7-2153-434b-8a1c-817eabac78b0-istiod-ca-cert\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.743892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.743867 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-credential-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.744002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.743920 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-certs\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.744274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.744253 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-credential-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.745650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.745624 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-envoy\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.745826 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.745807 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-podinfo\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.764154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.764126 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-token\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.764273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.764202 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9bw\" (UniqueName: \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-kube-api-access-dw9bw\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:43.855730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:43.855698 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:44.199815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:44.199785 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q"] Oct 30 13:30:44.202406 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:44.202379 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd91c2b7_2153_434b_8a1c_817eabac78b0.slice/crio-1c2796481a3ca27f7b03c4a12e353352b3dafb3fbb9442d15ee6e8145af183d9 WatchSource:0}: Error finding container 1c2796481a3ca27f7b03c4a12e353352b3dafb3fbb9442d15ee6e8145af183d9: Status 404 returned error can't find the container with id 1c2796481a3ca27f7b03c4a12e353352b3dafb3fbb9442d15ee6e8145af183d9 Oct 30 13:30:44.205607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:44.205573 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:44.205704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:44.205651 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:44.205704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:44.205696 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:44.327532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:44.327491 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" event={"ID":"fd91c2b7-2153-434b-8a1c-817eabac78b0","Type":"ContainerStarted","Data":"27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579"} Oct 30 13:30:44.327697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:44.327562 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" event={"ID":"fd91c2b7-2153-434b-8a1c-817eabac78b0","Type":"ContainerStarted","Data":"1c2796481a3ca27f7b03c4a12e353352b3dafb3fbb9442d15ee6e8145af183d9"} Oct 30 13:30:44.382167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:44.382115 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" podStartSLOduration=1.382100915 podStartE2EDuration="1.382100915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:44.376442024 +0000 UTC m=+2351.191636448" watchObservedRunningTime="2026-10-30 13:30:44.382100915 +0000 UTC m=+2351.197295335" Oct 30 13:30:44.856830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:44.856797 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:45.138888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.138802 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48"] Oct 30 13:30:45.143803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.143777 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.167467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.167440 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48"] Oct 30 13:30:45.167882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.167855 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2140b530-29d4-4965-8467-dbcabafe9b24-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.167937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.167899 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.167983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.167969 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm95q\" (UniqueName: \"kubernetes.io/projected/2140b530-29d4-4965-8467-dbcabafe9b24-kube-api-access-xm95q\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.168046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.168029 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.172617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.172594 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-82796d-serving-cert\"" Oct 30 13:30:45.242028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.241985 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq"] Oct 30 13:30:45.256665 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.256638 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.258885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.258856 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq"] Oct 30 13:30:45.269563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.269513 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.270078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.270046 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2140b530-29d4-4965-8467-dbcabafe9b24-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.270184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.270092 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.270184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.270119 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm95q\" (UniqueName: \"kubernetes.io/projected/2140b530-29d4-4965-8467-dbcabafe9b24-kube-api-access-xm95q\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.271359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.271294 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2140b530-29d4-4965-8467-dbcabafe9b24-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.271597 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:45.271578 5049 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-serving-cert: secret "maas-api-e2e-shared-b-w5-82796d-serving-cert" not found Oct 30 13:30:45.271752 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:45.271738 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-maas-api-tls podName:2140b530-29d4-4965-8467-dbcabafe9b24 nodeName:}" failed. No retries permitted until 2026-10-30 13:30:45.77171822 +0000 UTC m=+2352.586912627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-maas-api-tls") pod "maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" (UID: "2140b530-29d4-4965-8467-dbcabafe9b24") : secret "maas-api-e2e-shared-b-w5-82796d-serving-cert" not found Oct 30 13:30:45.272519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.272500 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.298341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.298288 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:30:45.298597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.298297 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:30:45.332415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.331592 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:45.338705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.338678 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm95q\" (UniqueName: \"kubernetes.io/projected/2140b530-29d4-4965-8467-dbcabafe9b24-kube-api-access-xm95q\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.371184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.371151 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbfql\" (UniqueName: \"kubernetes.io/projected/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-kube-api-access-zbfql\") pod \"maas-api-revoke-sub-12826b2b-lq4cq\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.371571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.371471 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12826b2b-lq4cq\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.371571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.371542 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12826b2b-lq4cq\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.473717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.473620 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12826b2b-lq4cq\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.473717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.473689 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12826b2b-lq4cq\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.474254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.473739 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbfql\" (UniqueName: \"kubernetes.io/projected/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-kube-api-access-zbfql\") pod \"maas-api-revoke-sub-12826b2b-lq4cq\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.475115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.475075 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12826b2b-lq4cq\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.476620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.476595 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12826b2b-lq4cq\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.496016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.495980 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbfql\" (UniqueName: \"kubernetes.io/projected/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-kube-api-access-zbfql\") pod \"maas-api-revoke-sub-12826b2b-lq4cq\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.529343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.528846 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:45.578677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.578641 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:45.780178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.779114 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.781867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.781830 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:45.857826 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.857779 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:30:45.858000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.857845 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" podUID="fd91c2b7-2153-434b-8a1c-817eabac78b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 13:30:45.936886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:45.936853 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq"] Oct 30 13:30:45.938522 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:45.938491 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f38bfbd_51c0_40e2_aad4_6feaf8df3a95.slice/crio-0ce5307816bb85572d682025fda522864580a8b1915b806d6dbbf897bc357aca WatchSource:0}: Error finding container 0ce5307816bb85572d682025fda522864580a8b1915b806d6dbbf897bc357aca: Status 404 returned error can't find the container with id 0ce5307816bb85572d682025fda522864580a8b1915b806d6dbbf897bc357aca Oct 30 13:30:46.029435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:46.029397 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:46.053234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:46.052354 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:46.060571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:46.060542 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:46.356341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:46.356245 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" event={"ID":"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95","Type":"ContainerStarted","Data":"0ce5307816bb85572d682025fda522864580a8b1915b806d6dbbf897bc357aca"} Oct 30 13:30:46.369010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:46.368984 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:30:46.468987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:46.468947 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48"] Oct 30 13:30:46.856462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:46.856429 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:30:46.856914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:46.856890 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" podUID="fd91c2b7-2153-434b-8a1c-817eabac78b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 13:30:47.366916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.366879 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" event={"ID":"2140b530-29d4-4965-8467-dbcabafe9b24","Type":"ContainerStarted","Data":"7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c"} Oct 30 13:30:47.367091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.366937 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" event={"ID":"2140b530-29d4-4965-8467-dbcabafe9b24","Type":"ContainerStarted","Data":"258d561b077059547055ec4c3f354292f41895c782d06b6fecd2bbc9ec682808"} Oct 30 13:30:47.367155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.367134 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:47.369095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.368908 5049 generic.go:356] "Generic (PLEG): container finished" podID="0f38bfbd-51c0-40e2-aad4-6feaf8df3a95" containerID="19a683d592ac06aaf5f05333c46c1e3fa1aae2d1be75a2a7bc506adac52fffc5" exitCode=0 Oct 30 13:30:47.369095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.369016 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" event={"ID":"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95","Type":"ContainerDied","Data":"19a683d592ac06aaf5f05333c46c1e3fa1aae2d1be75a2a7bc506adac52fffc5"} Oct 30 13:30:47.454516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.453835 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" podStartSLOduration=1.963105938 podStartE2EDuration="2.453819191s" totalImagesPullingTime="490.713253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:47.448956398 +0000 UTC m=+2354.264150841" watchObservedRunningTime="2026-10-30 13:30:47.453819191 +0000 UTC m=+2354.269013613" Oct 30 13:30:47.463970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.463923 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:47.679202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.679166 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj"] Oct 30 13:30:47.684623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.684385 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.685582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.685159 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:47.748252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.748162 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj"] Oct 30 13:30:47.764114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.764078 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.764273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.764119 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.764273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.764180 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wmr\" (UniqueName: \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-kube-api-access-98wmr\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.764273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.764235 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.764522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.764299 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.764522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.764408 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.764522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.764474 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.764680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.764559 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.764680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.764588 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.856457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.856426 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:30:47.856664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.856642 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" podUID="fd91c2b7-2153-434b-8a1c-817eabac78b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 13:30:47.866219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866189 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.866398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866258 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.866398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866278 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.866398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866387 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.866590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866406 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.866590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866449 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wmr\" (UniqueName: \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-kube-api-access-98wmr\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.866590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866487 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.866590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866534 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.866800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866596 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.866800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866656 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.866800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866678 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.867012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.866994 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.867438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.867411 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.867527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.867445 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.868919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.868896 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.869157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.869141 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.888435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.888405 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wmr\" (UniqueName: \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-kube-api-access-98wmr\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:47.888538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:47.888455 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:48.004783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.004696 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:48.135741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.125267 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:48.371494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.370566 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" podUID="d87d52c7-bb6f-417e-b214-8646eba00ff2" containerName="main" containerID="cri-o://7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955" gracePeriod=45 Oct 30 13:30:48.449295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.449263 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj"] Oct 30 13:30:48.480681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.480643 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:48.480820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.480716 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:48.480820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.480745 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:48.747344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.746783 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:48.859346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.859089 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:30:48.859346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.859147 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" podUID="fd91c2b7-2153-434b-8a1c-817eabac78b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 13:30:48.904465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.904436 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:30:48.911380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.905465 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-xks-ca\") pod \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " Oct 30 13:30:48.911380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.905680 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-kube-api-access-zbfql\" (UniqueName: \"kubernetes.io/projected/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-kube-api-access-zbfql\") pod \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " Oct 30 13:30:48.911380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.905757 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-openshift-ca\") pod \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\" (UID: \"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95\") " Oct 30 13:30:48.911855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.911828 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-openshift-ca" pod "0f38bfbd-51c0-40e2-aad4-6feaf8df3a95" (UID: "0f38bfbd-51c0-40e2-aad4-6feaf8df3a95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:48.945457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.945394 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-xks-ca" pod "0f38bfbd-51c0-40e2-aad4-6feaf8df3a95" (UID: "0f38bfbd-51c0-40e2-aad4-6feaf8df3a95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:30:48.945681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:48.945074 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-kube-api-access-zbfql" pod "0f38bfbd-51c0-40e2-aad4-6feaf8df3a95" (UID: "0f38bfbd-51c0-40e2-aad4-6feaf8df3a95"). InnerVolumeSpecName "kube-api-access-zbfql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.035168 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-tmp-dir\") pod \"d87d52c7-bb6f-417e-b214-8646eba00ff2\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.035223 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-home\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-home\") pod \"d87d52c7-bb6f-417e-b214-8646eba00ff2\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.035365 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-model-cache\") pod \"d87d52c7-bb6f-417e-b214-8646eba00ff2\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.035451 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87d52c7-bb6f-417e-b214-8646eba00ff2-kube-api-access-zdd9h\" (UniqueName: \"kubernetes.io/projected/d87d52c7-bb6f-417e-b214-8646eba00ff2-kube-api-access-zdd9h\") pod \"d87d52c7-bb6f-417e-b214-8646eba00ff2\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.035541 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d87d52c7-bb6f-417e-b214-8646eba00ff2-tls-certs\" (UniqueName: \"kubernetes.io/secret/d87d52c7-bb6f-417e-b214-8646eba00ff2-tls-certs\") pod \"d87d52c7-bb6f-417e-b214-8646eba00ff2\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.035758 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-dshm\") pod \"d87d52c7-bb6f-417e-b214-8646eba00ff2\" (UID: \"d87d52c7-bb6f-417e-b214-8646eba00ff2\") " Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.036487 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.036512 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbfql\" (UniqueName: \"kubernetes.io/projected/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-kube-api-access-zbfql\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.036529 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.038581 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-model-cache" pod "d87d52c7-bb6f-417e-b214-8646eba00ff2" (UID: "d87d52c7-bb6f-417e-b214-8646eba00ff2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.038794 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-tmp-dir" pod "d87d52c7-bb6f-417e-b214-8646eba00ff2" (UID: "d87d52c7-bb6f-417e-b214-8646eba00ff2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:49.041348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.039005 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-home" pod "d87d52c7-bb6f-417e-b214-8646eba00ff2" (UID: "d87d52c7-bb6f-417e-b214-8646eba00ff2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:49.051576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.051510 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-dshm" pod "d87d52c7-bb6f-417e-b214-8646eba00ff2" (UID: "d87d52c7-bb6f-417e-b214-8646eba00ff2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:49.052385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.052357 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87d52c7-bb6f-417e-b214-8646eba00ff2-kube-api-access-zdd9h" pod "d87d52c7-bb6f-417e-b214-8646eba00ff2" (UID: "d87d52c7-bb6f-417e-b214-8646eba00ff2"). InnerVolumeSpecName "kube-api-access-zdd9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:49.053048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.053026 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87d52c7-bb6f-417e-b214-8646eba00ff2-tls-certs" pod "d87d52c7-bb6f-417e-b214-8646eba00ff2" (UID: "d87d52c7-bb6f-417e-b214-8646eba00ff2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:30:49.145351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.144662 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87d52c7-bb6f-417e-b214-8646eba00ff2-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:49.145351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.144744 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:49.145351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.144763 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:49.145351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.144775 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:49.145351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.144790 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87d52c7-bb6f-417e-b214-8646eba00ff2-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:49.145351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.144805 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdd9h\" (UniqueName: \"kubernetes.io/projected/d87d52c7-bb6f-417e-b214-8646eba00ff2-kube-api-access-zdd9h\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:49.439346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.438919 5049 generic.go:356] "Generic (PLEG): container finished" podID="d87d52c7-bb6f-417e-b214-8646eba00ff2" containerID="7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955" exitCode=0 Oct 30 13:30:49.439346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.439229 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" event={"ID":"d87d52c7-bb6f-417e-b214-8646eba00ff2","Type":"ContainerDied","Data":"7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955"} Oct 30 13:30:49.439346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.439273 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" event={"ID":"d87d52c7-bb6f-417e-b214-8646eba00ff2","Type":"ContainerDied","Data":"e05ad9e40de9b9933efa41df9fe31c67c243276a3533a9fab0ef256bfcff3f5d"} Oct 30 13:30:49.439346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.439298 5049 scope.go:122] "RemoveContainer" containerID="7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955" Oct 30 13:30:49.439734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.439503 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" Oct 30 13:30:49.462463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.461894 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" event={"ID":"0f38bfbd-51c0-40e2-aad4-6feaf8df3a95","Type":"ContainerDied","Data":"0ce5307816bb85572d682025fda522864580a8b1915b806d6dbbf897bc357aca"} Oct 30 13:30:49.462463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.461969 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ce5307816bb85572d682025fda522864580a8b1915b806d6dbbf897bc357aca" Oct 30 13:30:49.462463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.462063 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq" Oct 30 13:30:49.476062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.476020 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" event={"ID":"4503b1a1-fb99-4ea3-9c7d-e124fb841276","Type":"ContainerStarted","Data":"f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7"} Oct 30 13:30:49.476222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.476122 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" event={"ID":"4503b1a1-fb99-4ea3-9c7d-e124fb841276","Type":"ContainerStarted","Data":"02f36e0faf50e00158f880cddaf83d235e16cc5e38952baa0ce608c5e779798c"} Oct 30 13:30:49.484962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.484936 5049 scope.go:122] "RemoveContainer" containerID="7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955" Oct 30 13:30:49.485380 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:49.485343 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955\": container with ID starting with 7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955 not found: ID does not exist" containerID="7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955" Oct 30 13:30:49.485676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.485589 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955"} err="failed to get container status \"7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955\": rpc error: code = NotFound desc = could not find container \"7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955\": container with ID starting with 7d67cea98994377ca2710b2c4cf131ac33e7ffdce5fb3e1ec37aa729594a2955 not found: ID does not exist" Oct 30 13:30:49.545369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.536410 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:49.583450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.583409 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j"] Oct 30 13:30:49.605108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.603039 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j"] Oct 30 13:30:49.631399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.630582 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" podStartSLOduration=2.630558045 podStartE2EDuration="2.630558045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:49.588953649 +0000 UTC m=+2356.404148073" watchObservedRunningTime="2026-10-30 13:30:49.630558045 +0000 UTC m=+2356.445752469" Oct 30 13:30:49.748346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.747460 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-cda0d9/auth-test-model-cda0d9-kserve-846bb7599c-4km8j" podUID="d87d52c7-bb6f-417e-b214-8646eba00ff2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.230:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:30:49.870348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.867787 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 13:30:49.870348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:49.867852 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" podUID="fd91c2b7-2153-434b-8a1c-817eabac78b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 13:30:50.005157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.005069 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:50.015904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.013823 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 13:30:50.015904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.013890 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 13:30:50.693803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.693736 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66"] Oct 30 13:30:50.696125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.696047 5049 cpu_manager.go:414] "containerMap: removing container" podUID="0f38bfbd-51c0-40e2-aad4-6feaf8df3a95" containerName="revoke-keys" Oct 30 13:30:50.696251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.696127 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f38bfbd-51c0-40e2-aad4-6feaf8df3a95" containerName="revoke-keys" Oct 30 13:30:50.696251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.696155 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d87d52c7-bb6f-417e-b214-8646eba00ff2" containerName="main" Oct 30 13:30:50.696251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.696165 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87d52c7-bb6f-417e-b214-8646eba00ff2" containerName="main" Oct 30 13:30:50.696466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.696439 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f38bfbd-51c0-40e2-aad4-6feaf8df3a95" containerName="revoke-keys" Oct 30 13:30:50.696466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.696464 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87d52c7-bb6f-417e-b214-8646eba00ff2" containerName="main" Oct 30 13:30:50.702394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.702372 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.760391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.749975 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66"] Oct 30 13:30:50.804364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.801475 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.804364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.801558 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6w8q\" (UniqueName: \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-kube-api-access-p6w8q\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.804364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.801613 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.804364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.801688 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.804364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.801722 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.804364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.801816 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.804364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.801840 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.804364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.801899 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.804364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.801923 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.903048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.903005 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6w8q\" (UniqueName: \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-kube-api-access-p6w8q\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.903547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.903096 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.903547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.903199 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.903547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.903238 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.903547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.903373 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.903547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.903397 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.903547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.903462 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.903547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.903487 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.903931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.903609 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.903931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.903612 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.904490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.904458 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.904610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.904531 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.904610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.904582 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.905218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.905192 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.947395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.936836 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.959666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.959633 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.959878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.959704 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6w8q\" (UniqueName: \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-kube-api-access-p6w8q\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:50.960917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:50.960128 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:51.006860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.006416 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 13:30:51.006860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.006475 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 13:30:51.051701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.051623 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:51.130346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.126852 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:51.130346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.128345 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:51.136340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.132738 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:30:51.365398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.358887 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:51.401655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.400848 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87d52c7-bb6f-417e-b214-8646eba00ff2" path="/var/lib/kubelet/pods/d87d52c7-bb6f-417e-b214-8646eba00ff2/volumes" Oct 30 13:30:51.462031 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:51.462003 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71949dbf_4b7a_45e7_bd4f_775596a65fbd.slice/crio-19f774031dd3d8300784991ce5957ad102dfa2f263f46e2f1e16c070bb1217dd WatchSource:0}: Error finding container 19f774031dd3d8300784991ce5957ad102dfa2f263f46e2f1e16c070bb1217dd: Status 404 returned error can't find the container with id 19f774031dd3d8300784991ce5957ad102dfa2f263f46e2f1e16c070bb1217dd Oct 30 13:30:51.464896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.464867 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66"] Oct 30 13:30:51.466000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.465965 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:51.466184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.466166 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:51.466308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.466291 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:51.499395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:51.499365 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" event={"ID":"71949dbf-4b7a-45e7-bd4f-775596a65fbd","Type":"ContainerStarted","Data":"19f774031dd3d8300784991ce5957ad102dfa2f263f46e2f1e16c070bb1217dd"} Oct 30 13:30:52.005233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:52.005199 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 13:30:52.005759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:52.005257 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 13:30:52.541481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:52.541440 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" event={"ID":"71949dbf-4b7a-45e7-bd4f-775596a65fbd","Type":"ContainerStarted","Data":"ef462c19b7fe24cb1635f6d5158a069948648bccef6ac007825776801f798487"} Oct 30 13:30:52.633345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:52.632507 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" podStartSLOduration=2.632486453 podStartE2EDuration="2.632486453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:52.59249923 +0000 UTC m=+2359.407693652" watchObservedRunningTime="2026-10-30 13:30:52.632486453 +0000 UTC m=+2359.447680877" Oct 30 13:30:53.005716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:53.005676 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 13:30:53.006369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:53.006294 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 13:30:53.052915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:53.052881 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:53.069510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:53.069465 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:30:53.069821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:53.069792 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 13:30:53.172720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:53.172683 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:53.403981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:53.403950 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:30:54.005602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:54.005564 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 13:30:54.005876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:54.005845 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 13:30:54.060767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:54.060728 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:30:54.061096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:54.061063 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 13:30:54.986999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:54.986962 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:55.005122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:55.005094 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 13:30:55.005267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:55.005144 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 13:30:55.055930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:55.055895 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:30:55.062003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:55.056595 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 13:30:56.043756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.043720 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:56.045099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.045072 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:56.045294 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.045265 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn"] Oct 30 13:30:56.050417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.050396 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:30:56.050534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.050473 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.052727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.052704 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:30:56.052828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.052758 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 13:30:56.067974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.067941 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:30:56.068263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.067955 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:30:56.072551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.072528 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn"] Oct 30 13:30:56.158405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.157312 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbmf\" (UniqueName: \"kubernetes.io/projected/35549aa7-a70b-419c-819b-aa5bdabc1036-kube-api-access-lmbmf\") pod \"maas-api-revoke-sub-366b8d90-8npfn\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.158405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.157599 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-366b8d90-8npfn\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.158405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.157637 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-366b8d90-8npfn\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.259755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.259715 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-366b8d90-8npfn\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.260032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.259994 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-366b8d90-8npfn\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.260425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.260390 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbmf\" (UniqueName: \"kubernetes.io/projected/35549aa7-a70b-419c-819b-aa5bdabc1036-kube-api-access-lmbmf\") pod \"maas-api-revoke-sub-366b8d90-8npfn\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.265111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.265083 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-366b8d90-8npfn\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.269726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.269666 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-366b8d90-8npfn\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.282852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.282804 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbmf\" (UniqueName: \"kubernetes.io/projected/35549aa7-a70b-419c-819b-aa5bdabc1036-kube-api-access-lmbmf\") pod \"maas-api-revoke-sub-366b8d90-8npfn\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.367605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.367517 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:56.681898 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:30:56.681814 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:30:56.744668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.744640 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn"] Oct 30 13:30:56.746595 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:30:56.746565 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35549aa7_a70b_419c_819b_aa5bdabc1036.slice/crio-0c0a3236172913e9b0636c08465341af2a13386674ca3a2d5fc16ea42c6b076e WatchSource:0}: Error finding container 0c0a3236172913e9b0636c08465341af2a13386674ca3a2d5fc16ea42c6b076e: Status 404 returned error can't find the container with id 0c0a3236172913e9b0636c08465341af2a13386674ca3a2d5fc16ea42c6b076e Oct 30 13:30:56.827343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:56.826856 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:57.053556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:57.053520 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:30:57.053732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:57.053577 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 13:30:57.683338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:57.683283 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" event={"ID":"35549aa7-a70b-419c-819b-aa5bdabc1036","Type":"ContainerStarted","Data":"3e15c9d0eddb32119dc7638ec0451a92f9ad3e934a947e73c3c65b535fa17eb7"} Oct 30 13:30:57.683837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:57.683382 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" event={"ID":"35549aa7-a70b-419c-819b-aa5bdabc1036","Type":"ContainerStarted","Data":"0c0a3236172913e9b0636c08465341af2a13386674ca3a2d5fc16ea42c6b076e"} Oct 30 13:30:57.746825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:57.746762 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" podStartSLOduration=1.7467416500000001 podStartE2EDuration="1.74674165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:57.74464969 +0000 UTC m=+2364.559844114" watchObservedRunningTime="2026-10-30 13:30:57.74674165 +0000 UTC m=+2364.561936075" Oct 30 13:30:58.040633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:58.040597 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj"] Oct 30 13:30:58.045943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:58.045916 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c90a6f86-hj4rj"] Oct 30 13:30:58.051962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:58.051937 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 13:30:58.052079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:58.051982 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 13:30:58.696724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:58.696687 5049 generic.go:356] "Generic (PLEG): container finished" podID="35549aa7-a70b-419c-819b-aa5bdabc1036" containerID="3e15c9d0eddb32119dc7638ec0451a92f9ad3e934a947e73c3c65b535fa17eb7" exitCode=0 Oct 30 13:30:58.697159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:58.696786 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" event={"ID":"35549aa7-a70b-419c-819b-aa5bdabc1036","Type":"ContainerDied","Data":"3e15c9d0eddb32119dc7638ec0451a92f9ad3e934a947e73c3c65b535fa17eb7"} Oct 30 13:30:58.788886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:58.784378 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:59.135407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.135372 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:59.136783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.136757 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:59.137853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.137831 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:30:59.358699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.358665 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f7af8e-8869-4195-a8be-10de819b2e5c" path="/var/lib/kubelet/pods/d7f7af8e-8869-4195-a8be-10de819b2e5c/volumes" Oct 30 13:30:59.878575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.878549 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:30:59.918260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.918225 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-openshift-ca\") pod \"35549aa7-a70b-419c-819b-aa5bdabc1036\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " Oct 30 13:30:59.918493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.918371 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35549aa7-a70b-419c-819b-aa5bdabc1036-kube-api-access-lmbmf\" (UniqueName: \"kubernetes.io/projected/35549aa7-a70b-419c-819b-aa5bdabc1036-kube-api-access-lmbmf\") pod \"35549aa7-a70b-419c-819b-aa5bdabc1036\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " Oct 30 13:30:59.918493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.918486 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-xks-ca\") pod \"35549aa7-a70b-419c-819b-aa5bdabc1036\" (UID: \"35549aa7-a70b-419c-819b-aa5bdabc1036\") " Oct 30 13:30:59.918696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.918669 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-openshift-ca" pod "35549aa7-a70b-419c-819b-aa5bdabc1036" (UID: "35549aa7-a70b-419c-819b-aa5bdabc1036"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:59.919244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.919197 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:30:59.922999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.922962 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35549aa7-a70b-419c-819b-aa5bdabc1036-kube-api-access-lmbmf" pod "35549aa7-a70b-419c-819b-aa5bdabc1036" (UID: "35549aa7-a70b-419c-819b-aa5bdabc1036"). InnerVolumeSpecName "kube-api-access-lmbmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:59.927499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:30:59.927463 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-xks-ca" pod "35549aa7-a70b-419c-819b-aa5bdabc1036" (UID: "35549aa7-a70b-419c-819b-aa5bdabc1036"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:31:00.020517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:00.020418 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbmf\" (UniqueName: \"kubernetes.io/projected/35549aa7-a70b-419c-819b-aa5bdabc1036-kube-api-access-lmbmf\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:00.020517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:00.020452 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35549aa7-a70b-419c-819b-aa5bdabc1036-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:00.583289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:00.583231 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:00.716097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:00.716063 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" Oct 30 13:31:00.716309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:00.716091 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn" event={"ID":"35549aa7-a70b-419c-819b-aa5bdabc1036","Type":"ContainerDied","Data":"0c0a3236172913e9b0636c08465341af2a13386674ca3a2d5fc16ea42c6b076e"} Oct 30 13:31:00.716309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:00.716146 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c0a3236172913e9b0636c08465341af2a13386674ca3a2d5fc16ea42c6b076e" Oct 30 13:31:02.282970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:02.282937 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:04.347657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:04.347619 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:06.141351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.140987 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:06.618890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.618846 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c"] Oct 30 13:31:06.620917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.620882 5049 cpu_manager.go:414] "containerMap: removing container" podUID="35549aa7-a70b-419c-819b-aa5bdabc1036" containerName="revoke-keys" Oct 30 13:31:06.620917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.620917 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35549aa7-a70b-419c-819b-aa5bdabc1036" containerName="revoke-keys" Oct 30 13:31:06.621164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.621145 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="35549aa7-a70b-419c-819b-aa5bdabc1036" containerName="revoke-keys" Oct 30 13:31:06.630230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.630199 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:06.644431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.644397 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c"] Oct 30 13:31:06.647766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.647740 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:31:06.648835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.648749 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:31:06.729419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.729380 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb64431-k2f2c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:06.729712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.729691 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgv6m\" (UniqueName: \"kubernetes.io/projected/eb21d7cf-6717-4154-911b-86802f32fd4c-kube-api-access-zgv6m\") pod \"maas-api-revoke-sub-8fb64431-k2f2c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:06.730316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.730290 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb64431-k2f2c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:06.831852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.831809 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb64431-k2f2c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:06.832065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.831953 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb64431-k2f2c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:06.832065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.832010 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgv6m\" (UniqueName: \"kubernetes.io/projected/eb21d7cf-6717-4154-911b-86802f32fd4c-kube-api-access-zgv6m\") pod \"maas-api-revoke-sub-8fb64431-k2f2c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:06.832802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.832773 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb64431-k2f2c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:06.834298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.834275 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb64431-k2f2c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:06.853675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.853647 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgv6m\" (UniqueName: \"kubernetes.io/projected/eb21d7cf-6717-4154-911b-86802f32fd4c-kube-api-access-zgv6m\") pod \"maas-api-revoke-sub-8fb64431-k2f2c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:06.954363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:06.954271 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:07.278157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:07.278105 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c"] Oct 30 13:31:07.282045 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:31:07.282017 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb21d7cf_6717_4154_911b_86802f32fd4c.slice/crio-a31f3df66c1f4c4b892b387430fb81954a9873b447beba8bd5d267538a10f120 WatchSource:0}: Error finding container a31f3df66c1f4c4b892b387430fb81954a9873b447beba8bd5d267538a10f120: Status 404 returned error can't find the container with id a31f3df66c1f4c4b892b387430fb81954a9873b447beba8bd5d267538a10f120 Oct 30 13:31:07.790044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:07.790001 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" event={"ID":"eb21d7cf-6717-4154-911b-86802f32fd4c","Type":"ContainerStarted","Data":"a31f3df66c1f4c4b892b387430fb81954a9873b447beba8bd5d267538a10f120"} Oct 30 13:31:07.975935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:07.975847 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:08.045080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:08.045037 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r"] Oct 30 13:31:08.053958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:08.053925 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99802cce-r8d8r"] Oct 30 13:31:08.867078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:08.844777 5049 generic.go:356] "Generic (PLEG): container finished" podID="eb21d7cf-6717-4154-911b-86802f32fd4c" containerID="3dab0f8620dcb8f9236e909308f10e4cf5064b3a5d37660834f67a6e9678b94f" exitCode=0 Oct 30 13:31:08.867078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:08.844944 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" event={"ID":"eb21d7cf-6717-4154-911b-86802f32fd4c","Type":"ContainerDied","Data":"3dab0f8620dcb8f9236e909308f10e4cf5064b3a5d37660834f67a6e9678b94f"} Oct 30 13:31:09.386344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.386224 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943b7f36-93c3-4815-8904-1e545470612d" path="/var/lib/kubelet/pods/943b7f36-93c3-4815-8904-1e545470612d/volumes" Oct 30 13:31:09.791070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.790988 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8"] Oct 30 13:31:09.797476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.797450 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.821595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.821545 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-9c6033\"/\"openshift-service-ca.crt\"" Oct 30 13:31:09.821595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.821575 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-9c6033\"/\"default-dockercfg-xqqj2\"" Oct 30 13:31:09.821780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.821654 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-9c6033\"/\"kube-root-ca.crt\"" Oct 30 13:31:09.821829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.821815 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-9c6033\"/\"rate-test-model-9c6033-kserve-self-signed-certs\"" Oct 30 13:31:09.828023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.827995 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8"] Oct 30 13:31:09.843828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.843795 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:09.851590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.851558 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-tmp-dir\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.851879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.851854 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-dshm\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.851953 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.851920 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-home\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.852023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.851990 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-model-cache\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.852089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.852067 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16efac8f-7d97-4b48-95d5-b1e8ed329020-tls-certs\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.852149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.852129 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkqcs\" (UniqueName: \"kubernetes.io/projected/16efac8f-7d97-4b48-95d5-b1e8ed329020-kube-api-access-kkqcs\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.954475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.953765 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-tmp-dir\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.954475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.954221 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-tmp-dir\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.955002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.954647 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-dshm\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.958527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.955090 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-home\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.958527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.955427 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-home\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.958527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.955996 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-model-cache\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.958527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.955681 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-model-cache\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.958527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.956164 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16efac8f-7d97-4b48-95d5-b1e8ed329020-tls-certs\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.958527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.956712 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqcs\" (UniqueName: \"kubernetes.io/projected/16efac8f-7d97-4b48-95d5-b1e8ed329020-kube-api-access-kkqcs\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.958527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.957559 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-dshm\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.959778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.959756 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16efac8f-7d97-4b48-95d5-b1e8ed329020-tls-certs\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:09.982849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:09.982819 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqcs\" (UniqueName: \"kubernetes.io/projected/16efac8f-7d97-4b48-95d5-b1e8ed329020-kube-api-access-kkqcs\") pod \"rate-test-model-9c6033-kserve-5b997cf465-tt5t8\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:10.004013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.003972 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:10.079943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.079912 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:10.113825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.113794 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:10.198383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.184662 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-xks-ca\") pod \"eb21d7cf-6717-4154-911b-86802f32fd4c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " Oct 30 13:31:10.198383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.184751 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb21d7cf-6717-4154-911b-86802f32fd4c-kube-api-access-zgv6m\" (UniqueName: \"kubernetes.io/projected/eb21d7cf-6717-4154-911b-86802f32fd4c-kube-api-access-zgv6m\") pod \"eb21d7cf-6717-4154-911b-86802f32fd4c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " Oct 30 13:31:10.198383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.184784 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-openshift-ca\") pod \"eb21d7cf-6717-4154-911b-86802f32fd4c\" (UID: \"eb21d7cf-6717-4154-911b-86802f32fd4c\") " Oct 30 13:31:10.198383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.185870 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-openshift-ca" pod "eb21d7cf-6717-4154-911b-86802f32fd4c" (UID: "eb21d7cf-6717-4154-911b-86802f32fd4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:31:10.198383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.188935 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-xks-ca" pod "eb21d7cf-6717-4154-911b-86802f32fd4c" (UID: "eb21d7cf-6717-4154-911b-86802f32fd4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:31:10.224529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.221525 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb21d7cf-6717-4154-911b-86802f32fd4c-kube-api-access-zgv6m" pod "eb21d7cf-6717-4154-911b-86802f32fd4c" (UID: "eb21d7cf-6717-4154-911b-86802f32fd4c"). InnerVolumeSpecName "kube-api-access-zgv6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:31:10.287607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.287569 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgv6m\" (UniqueName: \"kubernetes.io/projected/eb21d7cf-6717-4154-911b-86802f32fd4c-kube-api-access-zgv6m\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:10.287607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.287608 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:10.287817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.287626 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb21d7cf-6717-4154-911b-86802f32fd4c-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:10.467280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.467255 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8"] Oct 30 13:31:10.472402 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:31:10.471596 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16efac8f_7d97_4b48_95d5_b1e8ed329020.slice/crio-36d875837e8f9f2b03ec72ef4459aca1932c2d605f5595eb1db4c3355c1d7ddd WatchSource:0}: Error finding container 36d875837e8f9f2b03ec72ef4459aca1932c2d605f5595eb1db4c3355c1d7ddd: Status 404 returned error can't find the container with id 36d875837e8f9f2b03ec72ef4459aca1932c2d605f5595eb1db4c3355c1d7ddd Oct 30 13:31:10.877448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.877413 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" Oct 30 13:31:10.877654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.877410 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c" event={"ID":"eb21d7cf-6717-4154-911b-86802f32fd4c","Type":"ContainerDied","Data":"a31f3df66c1f4c4b892b387430fb81954a9873b447beba8bd5d267538a10f120"} Oct 30 13:31:10.877654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.877572 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31f3df66c1f4c4b892b387430fb81954a9873b447beba8bd5d267538a10f120" Oct 30 13:31:10.879496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.879461 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" event={"ID":"16efac8f-7d97-4b48-95d5-b1e8ed329020","Type":"ContainerStarted","Data":"42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0"} Oct 30 13:31:10.879622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.879506 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" event={"ID":"16efac8f-7d97-4b48-95d5-b1e8ed329020","Type":"ContainerStarted","Data":"36d875837e8f9f2b03ec72ef4459aca1932c2d605f5595eb1db4c3355c1d7ddd"} Oct 30 13:31:10.921337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:10.921279 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" podStartSLOduration=1.9212654850000002 podStartE2EDuration="1.921265485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:10.917501337 +0000 UTC m=+2377.732695762" watchObservedRunningTime="2026-10-30 13:31:10.921265485 +0000 UTC m=+2377.736459949" Oct 30 13:31:11.114677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:11.114636 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:11.825261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:11.825225 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:13.507186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.507151 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk"] Oct 30 13:31:13.507964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.507948 5049 cpu_manager.go:414] "containerMap: removing container" podUID="eb21d7cf-6717-4154-911b-86802f32fd4c" containerName="revoke-keys" Oct 30 13:31:13.508024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.507966 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb21d7cf-6717-4154-911b-86802f32fd4c" containerName="revoke-keys" Oct 30 13:31:13.508109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.508099 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb21d7cf-6717-4154-911b-86802f32fd4c" containerName="revoke-keys" Oct 30 13:31:13.512058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.512041 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:13.526229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.526207 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:31:13.538349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.536307 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:31:13.546361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.542644 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk"] Oct 30 13:31:13.638821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.638776 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4xhn\" (UniqueName: \"kubernetes.io/projected/754e84ac-3785-4b17-962c-f555525caf40-kube-api-access-h4xhn\") pod \"maas-api-revoke-sub-6c746f1a-rw8wk\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:13.639033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.638909 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754e84ac-3785-4b17-962c-f555525caf40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c746f1a-rw8wk\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:13.639033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.638997 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754e84ac-3785-4b17-962c-f555525caf40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c746f1a-rw8wk\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:13.741007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.740721 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4xhn\" (UniqueName: \"kubernetes.io/projected/754e84ac-3785-4b17-962c-f555525caf40-kube-api-access-h4xhn\") pod \"maas-api-revoke-sub-6c746f1a-rw8wk\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:13.741007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.740838 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754e84ac-3785-4b17-962c-f555525caf40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c746f1a-rw8wk\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:13.741007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.740922 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754e84ac-3785-4b17-962c-f555525caf40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c746f1a-rw8wk\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:13.742574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.742299 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754e84ac-3785-4b17-962c-f555525caf40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c746f1a-rw8wk\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:13.745546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.745487 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754e84ac-3785-4b17-962c-f555525caf40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c746f1a-rw8wk\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:13.755387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.751356 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:13.765774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.765693 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4xhn\" (UniqueName: \"kubernetes.io/projected/754e84ac-3785-4b17-962c-f555525caf40-kube-api-access-h4xhn\") pod \"maas-api-revoke-sub-6c746f1a-rw8wk\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:13.829423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:13.829393 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:14.145199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:14.145173 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk"] Oct 30 13:31:14.148153 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:31:14.148122 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754e84ac_3785_4b17_962c_f555525caf40.slice/crio-857b685b5dba9ba1aa497157346c5fd3ee8dfb332220b0e86b67d5bfa1df1341 WatchSource:0}: Error finding container 857b685b5dba9ba1aa497157346c5fd3ee8dfb332220b0e86b67d5bfa1df1341: Status 404 returned error can't find the container with id 857b685b5dba9ba1aa497157346c5fd3ee8dfb332220b0e86b67d5bfa1df1341 Oct 30 13:31:14.944812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:14.944773 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" event={"ID":"754e84ac-3785-4b17-962c-f555525caf40","Type":"ContainerStarted","Data":"3daba02c4f0f91e2d637091db35046eaeefe586660ad756895b12a5894e167be"} Oct 30 13:31:14.945358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:14.944861 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" event={"ID":"754e84ac-3785-4b17-962c-f555525caf40","Type":"ContainerStarted","Data":"857b685b5dba9ba1aa497157346c5fd3ee8dfb332220b0e86b67d5bfa1df1341"} Oct 30 13:31:14.981858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:14.981803 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" podStartSLOduration=1.9817906060000001 podStartE2EDuration="1.981790606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:14.981346224 +0000 UTC m=+2381.796540639" watchObservedRunningTime="2026-10-30 13:31:14.981790606 +0000 UTC m=+2381.796985029" Oct 30 13:31:15.735371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:15.734197 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:15.955662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:15.955623 5049 generic.go:356] "Generic (PLEG): container finished" podID="754e84ac-3785-4b17-962c-f555525caf40" containerID="3daba02c4f0f91e2d637091db35046eaeefe586660ad756895b12a5894e167be" exitCode=0 Oct 30 13:31:15.956067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:15.955703 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" event={"ID":"754e84ac-3785-4b17-962c-f555525caf40","Type":"ContainerDied","Data":"3daba02c4f0f91e2d637091db35046eaeefe586660ad756895b12a5894e167be"} Oct 30 13:31:17.136458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.136423 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:17.187928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.187889 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/754e84ac-3785-4b17-962c-f555525caf40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754e84ac-3785-4b17-962c-f555525caf40-maas-api-xks-ca\") pod \"754e84ac-3785-4b17-962c-f555525caf40\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " Oct 30 13:31:17.188139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.188062 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754e84ac-3785-4b17-962c-f555525caf40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754e84ac-3785-4b17-962c-f555525caf40-maas-api-openshift-ca\") pod \"754e84ac-3785-4b17-962c-f555525caf40\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " Oct 30 13:31:17.188222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.188193 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754e84ac-3785-4b17-962c-f555525caf40-kube-api-access-h4xhn\" (UniqueName: \"kubernetes.io/projected/754e84ac-3785-4b17-962c-f555525caf40-kube-api-access-h4xhn\") pod \"754e84ac-3785-4b17-962c-f555525caf40\" (UID: \"754e84ac-3785-4b17-962c-f555525caf40\") " Oct 30 13:31:17.188981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.188925 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754e84ac-3785-4b17-962c-f555525caf40-maas-api-openshift-ca" pod "754e84ac-3785-4b17-962c-f555525caf40" (UID: "754e84ac-3785-4b17-962c-f555525caf40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:31:17.194978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.190888 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/754e84ac-3785-4b17-962c-f555525caf40-maas-api-xks-ca" pod "754e84ac-3785-4b17-962c-f555525caf40" (UID: "754e84ac-3785-4b17-962c-f555525caf40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:31:17.194978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.191230 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754e84ac-3785-4b17-962c-f555525caf40-kube-api-access-h4xhn" pod "754e84ac-3785-4b17-962c-f555525caf40" (UID: "754e84ac-3785-4b17-962c-f555525caf40"). InnerVolumeSpecName "kube-api-access-h4xhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:31:17.290260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.290178 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754e84ac-3785-4b17-962c-f555525caf40-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:17.290260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.290209 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754e84ac-3785-4b17-962c-f555525caf40-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:17.290260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.290218 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4xhn\" (UniqueName: \"kubernetes.io/projected/754e84ac-3785-4b17-962c-f555525caf40-kube-api-access-h4xhn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:17.556700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.556642 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:17.977640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.977554 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" Oct 30 13:31:17.977640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.977581 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk" event={"ID":"754e84ac-3785-4b17-962c-f555525caf40","Type":"ContainerDied","Data":"857b685b5dba9ba1aa497157346c5fd3ee8dfb332220b0e86b67d5bfa1df1341"} Oct 30 13:31:17.977640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:17.977631 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="857b685b5dba9ba1aa497157346c5fd3ee8dfb332220b0e86b67d5bfa1df1341" Oct 30 13:31:19.502084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:19.502027 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:20.114193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:20.114156 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:20.126704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:20.126675 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:21.016368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:21.016333 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:31:21.043956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:21.043920 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l"] Oct 30 13:31:21.048250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:21.048221 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b6092f3-ffr4l"] Oct 30 13:31:21.338967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:21.338932 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e392228-99d1-4bc8-9567-a293c9e7a20a" path="/var/lib/kubelet/pods/4e392228-99d1-4bc8-9567-a293c9e7a20a/volumes" Oct 30 13:31:21.410396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:21.410360 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:21.965591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:21.965556 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q"] Oct 30 13:31:22.047813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:22.047776 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:23.246351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:23.246287 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:23.836875 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:23.836836 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk"] Oct 30 13:31:23.837652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:23.837633 5049 cpu_manager.go:414] "containerMap: removing container" podUID="754e84ac-3785-4b17-962c-f555525caf40" containerName="revoke-keys" Oct 30 13:31:23.837652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:23.837652 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754e84ac-3785-4b17-962c-f555525caf40" containerName="revoke-keys" Oct 30 13:31:23.837825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:23.837806 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="754e84ac-3785-4b17-962c-f555525caf40" containerName="revoke-keys" Oct 30 13:31:23.843278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:23.843256 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:23.863838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:23.863806 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk"] Oct 30 13:31:23.868509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:23.868489 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:31:23.868659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:23.868643 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:31:24.007072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.007036 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrm2\" (UniqueName: \"kubernetes.io/projected/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-kube-api-access-pmrm2\") pod \"maas-api-revoke-sub-33efb373-j9whk\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:24.007356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.007307 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33efb373-j9whk\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:24.007581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.007561 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33efb373-j9whk\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:24.109086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.108986 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33efb373-j9whk\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:24.109086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.109070 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrm2\" (UniqueName: \"kubernetes.io/projected/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-kube-api-access-pmrm2\") pod \"maas-api-revoke-sub-33efb373-j9whk\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:24.109356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.109251 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33efb373-j9whk\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:24.109857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.109827 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33efb373-j9whk\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:24.111986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.111958 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33efb373-j9whk\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:24.131421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.131392 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrm2\" (UniqueName: \"kubernetes.io/projected/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-kube-api-access-pmrm2\") pod \"maas-api-revoke-sub-33efb373-j9whk\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:24.158458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.158431 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:24.471549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.471518 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk"] Oct 30 13:31:24.473565 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:31:24.473535 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273bd15c_ba75_4eca_8fa6_3dfea8324ee3.slice/crio-21430b799b5b46f2ed880e47380c10edaee77699500756ea38b82c36324bddbc WatchSource:0}: Error finding container 21430b799b5b46f2ed880e47380c10edaee77699500756ea38b82c36324bddbc: Status 404 returned error can't find the container with id 21430b799b5b46f2ed880e47380c10edaee77699500756ea38b82c36324bddbc Oct 30 13:31:24.774885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:24.774854 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:25.051789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:25.051747 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" event={"ID":"273bd15c-ba75-4eca-8fa6-3dfea8324ee3","Type":"ContainerStarted","Data":"ffed352db45683e2cdd96aa9b4d60c8169fab78c70b7f8a27cb7294e24425de3"} Oct 30 13:31:25.051973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:25.051819 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" event={"ID":"273bd15c-ba75-4eca-8fa6-3dfea8324ee3","Type":"ContainerStarted","Data":"21430b799b5b46f2ed880e47380c10edaee77699500756ea38b82c36324bddbc"} Oct 30 13:31:25.090608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:25.090552 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" podStartSLOduration=2.090536145 podStartE2EDuration="2.090536145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:25.088128197 +0000 UTC m=+2391.903322620" watchObservedRunningTime="2026-10-30 13:31:25.090536145 +0000 UTC m=+2391.905730567" Oct 30 13:31:25.107855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:25.107818 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:26.062501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:26.062468 5049 generic.go:356] "Generic (PLEG): container finished" podID="273bd15c-ba75-4eca-8fa6-3dfea8324ee3" containerID="ffed352db45683e2cdd96aa9b4d60c8169fab78c70b7f8a27cb7294e24425de3" exitCode=0 Oct 30 13:31:26.062925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:26.062554 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" event={"ID":"273bd15c-ba75-4eca-8fa6-3dfea8324ee3","Type":"ContainerDied","Data":"ffed352db45683e2cdd96aa9b4d60c8169fab78c70b7f8a27cb7294e24425de3"} Oct 30 13:31:26.679905 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:31:26.679867 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:31:27.257226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.257196 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:27.267816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.267694 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:27.277712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.277044 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-kube-api-access-pmrm2\" (UniqueName: \"kubernetes.io/projected/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-kube-api-access-pmrm2\") pod \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " Oct 30 13:31:27.277712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.277145 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-xks-ca\") pod \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " Oct 30 13:31:27.277712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.277388 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-openshift-ca\") pod \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\" (UID: \"273bd15c-ba75-4eca-8fa6-3dfea8324ee3\") " Oct 30 13:31:27.278237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.278209 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-openshift-ca" pod "273bd15c-ba75-4eca-8fa6-3dfea8324ee3" (UID: "273bd15c-ba75-4eca-8fa6-3dfea8324ee3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:31:27.280408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.280379 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-xks-ca" pod "273bd15c-ba75-4eca-8fa6-3dfea8324ee3" (UID: "273bd15c-ba75-4eca-8fa6-3dfea8324ee3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:31:27.280549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.280524 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-kube-api-access-pmrm2" pod "273bd15c-ba75-4eca-8fa6-3dfea8324ee3" (UID: "273bd15c-ba75-4eca-8fa6-3dfea8324ee3"). InnerVolumeSpecName "kube-api-access-pmrm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:31:27.379823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.379792 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:27.379823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.379820 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrm2\" (UniqueName: \"kubernetes.io/projected/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-kube-api-access-pmrm2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:27.379823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:27.379830 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273bd15c-ba75-4eca-8fa6-3dfea8324ee3-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:28.081612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:28.081571 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" event={"ID":"273bd15c-ba75-4eca-8fa6-3dfea8324ee3","Type":"ContainerDied","Data":"21430b799b5b46f2ed880e47380c10edaee77699500756ea38b82c36324bddbc"} Oct 30 13:31:28.081794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:28.081638 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21430b799b5b46f2ed880e47380c10edaee77699500756ea38b82c36324bddbc" Oct 30 13:31:28.081794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:28.081592 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk" Oct 30 13:31:28.702500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:28.702471 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:29.128986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:29.128866 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:31.072128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:31.072092 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:32.628344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:32.615171 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:32.816640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:32.816596 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:33.018138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:33.018058 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:33.790674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:33.790639 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:33.839688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:33.839660 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:31:33.843681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:33.843650 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:31:34.233134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.233085 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:34.725829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.725791 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:34.762137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.762108 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx"] Oct 30 13:31:34.762883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.762870 5049 cpu_manager.go:414] "containerMap: removing container" podUID="273bd15c-ba75-4eca-8fa6-3dfea8324ee3" containerName="revoke-keys" Oct 30 13:31:34.762930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.762886 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273bd15c-ba75-4eca-8fa6-3dfea8324ee3" containerName="revoke-keys" Oct 30 13:31:34.763048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.763039 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="273bd15c-ba75-4eca-8fa6-3dfea8324ee3" containerName="revoke-keys" Oct 30 13:31:34.768583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.768554 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:34.783056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.783029 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:31:34.793169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.793144 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:31:34.799575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.799540 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx"] Oct 30 13:31:34.900343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.898544 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:34.938933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.938894 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f443e7c-5xbkx\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:34.956060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.956027 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnxl\" (UniqueName: \"kubernetes.io/projected/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-kube-api-access-qgnxl\") pod \"maas-api-revoke-sub-1f443e7c-5xbkx\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:34.973657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:34.973618 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f443e7c-5xbkx\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:35.074987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:35.074951 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f443e7c-5xbkx\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:35.074987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:35.074992 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnxl\" (UniqueName: \"kubernetes.io/projected/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-kube-api-access-qgnxl\") pod \"maas-api-revoke-sub-1f443e7c-5xbkx\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:35.075276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:35.075090 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f443e7c-5xbkx\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:35.075802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:35.075775 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f443e7c-5xbkx\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:35.077440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:35.077414 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f443e7c-5xbkx\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:35.098673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:35.098646 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnxl\" (UniqueName: \"kubernetes.io/projected/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-kube-api-access-qgnxl\") pod \"maas-api-revoke-sub-1f443e7c-5xbkx\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:35.388142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:35.388065 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:35.708867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:35.708835 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx"] Oct 30 13:31:35.709453 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:31:35.709426 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f1b9ca_dcd0_411b_b2c3_cee56accd687.slice/crio-c873a51a3cb547bc12d43f85e74d4a4929a6e115ee24357bed9ff8c54281cb03 WatchSource:0}: Error finding container c873a51a3cb547bc12d43f85e74d4a4929a6e115ee24357bed9ff8c54281cb03: Status 404 returned error can't find the container with id c873a51a3cb547bc12d43f85e74d4a4929a6e115ee24357bed9ff8c54281cb03 Oct 30 13:31:36.161571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:36.161528 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" event={"ID":"d9f1b9ca-dcd0-411b-b2c3-cee56accd687","Type":"ContainerStarted","Data":"4e67821c9ffaf97dba269bea0c8bc4eed606b421a7eae56a3a24be7ab2b7395e"} Oct 30 13:31:36.162008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:36.161593 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" event={"ID":"d9f1b9ca-dcd0-411b-b2c3-cee56accd687","Type":"ContainerStarted","Data":"c873a51a3cb547bc12d43f85e74d4a4929a6e115ee24357bed9ff8c54281cb03"} Oct 30 13:31:36.164277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:36.164253 5049 scope.go:122] "RemoveContainer" containerID="ad2facc231d6a7fd8736eadd5dba460444a7e7dd65ed8145abf22e1b1c4f9147" Oct 30 13:31:36.176794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:36.176773 5049 scope.go:122] "RemoveContainer" containerID="113399acf81ecb74a9f23a3ba314bde3e2f8bbd0136459329b764ad281b8cd89" Oct 30 13:31:36.188328 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:36.188294 5049 scope.go:122] "RemoveContainer" containerID="afd5d1e8b389b6ecd2ac1117b3b8ccc6d47daca0612fa6e1e9d8ed56eef52648" Oct 30 13:31:36.203757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:36.203706 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" podStartSLOduration=2.203690196 podStartE2EDuration="2.203690196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:36.198646401 +0000 UTC m=+2403.013840821" watchObservedRunningTime="2026-10-30 13:31:36.203690196 +0000 UTC m=+2403.018884619" Oct 30 13:31:36.955831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:36.955688 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:36.995346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:36.991620 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" podUID="2458d771-9f41-48fd-9e75-78f4774c278b" containerName="main" containerID="cri-o://b5e38bee59081af1c7757b20e2e94cf7ad12cc4b04db92d681f69a1b85940c25" gracePeriod=45 Oct 30 13:31:37.172886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.172718 5049 generic.go:356] "Generic (PLEG): container finished" podID="d9f1b9ca-dcd0-411b-b2c3-cee56accd687" containerID="4e67821c9ffaf97dba269bea0c8bc4eed606b421a7eae56a3a24be7ab2b7395e" exitCode=0 Oct 30 13:31:37.172886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.172804 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" event={"ID":"d9f1b9ca-dcd0-411b-b2c3-cee56accd687","Type":"ContainerDied","Data":"4e67821c9ffaf97dba269bea0c8bc4eed606b421a7eae56a3a24be7ab2b7395e"} Oct 30 13:31:37.175094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.175068 5049 generic.go:356] "Generic (PLEG): container finished" podID="2458d771-9f41-48fd-9e75-78f4774c278b" containerID="b5e38bee59081af1c7757b20e2e94cf7ad12cc4b04db92d681f69a1b85940c25" exitCode=0 Oct 30 13:31:37.175213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.175173 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" event={"ID":"2458d771-9f41-48fd-9e75-78f4774c278b","Type":"ContainerDied","Data":"b5e38bee59081af1c7757b20e2e94cf7ad12cc4b04db92d681f69a1b85940c25"} Oct 30 13:31:37.349920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.349885 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:31:37.429069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.429036 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-home\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-home\") pod \"2458d771-9f41-48fd-9e75-78f4774c278b\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " Oct 30 13:31:37.429271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.429188 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2458d771-9f41-48fd-9e75-78f4774c278b-tls-certs\" (UniqueName: \"kubernetes.io/secret/2458d771-9f41-48fd-9e75-78f4774c278b-tls-certs\") pod \"2458d771-9f41-48fd-9e75-78f4774c278b\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " Oct 30 13:31:37.429271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.429242 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-dshm\") pod \"2458d771-9f41-48fd-9e75-78f4774c278b\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " Oct 30 13:31:37.429502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.429306 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-model-cache\") pod \"2458d771-9f41-48fd-9e75-78f4774c278b\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " Oct 30 13:31:37.429502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.429424 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-home" pod "2458d771-9f41-48fd-9e75-78f4774c278b" (UID: "2458d771-9f41-48fd-9e75-78f4774c278b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:31:37.429626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.429498 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2458d771-9f41-48fd-9e75-78f4774c278b-kube-api-access-jwdg5\" (UniqueName: \"kubernetes.io/projected/2458d771-9f41-48fd-9e75-78f4774c278b-kube-api-access-jwdg5\") pod \"2458d771-9f41-48fd-9e75-78f4774c278b\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " Oct 30 13:31:37.429698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.429622 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-tmp-dir\") pod \"2458d771-9f41-48fd-9e75-78f4774c278b\" (UID: \"2458d771-9f41-48fd-9e75-78f4774c278b\") " Oct 30 13:31:37.429698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.429643 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-model-cache" pod "2458d771-9f41-48fd-9e75-78f4774c278b" (UID: "2458d771-9f41-48fd-9e75-78f4774c278b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:31:37.429946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.429917 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-tmp-dir" pod "2458d771-9f41-48fd-9e75-78f4774c278b" (UID: "2458d771-9f41-48fd-9e75-78f4774c278b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:31:37.430844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.430775 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:37.430844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.430803 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:37.430844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.430821 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:37.431935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.431911 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2458d771-9f41-48fd-9e75-78f4774c278b-tls-certs" pod "2458d771-9f41-48fd-9e75-78f4774c278b" (UID: "2458d771-9f41-48fd-9e75-78f4774c278b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:31:37.433368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.433294 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2458d771-9f41-48fd-9e75-78f4774c278b-kube-api-access-jwdg5" pod "2458d771-9f41-48fd-9e75-78f4774c278b" (UID: "2458d771-9f41-48fd-9e75-78f4774c278b"). InnerVolumeSpecName "kube-api-access-jwdg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:31:37.433509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.433409 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-dshm" pod "2458d771-9f41-48fd-9e75-78f4774c278b" (UID: "2458d771-9f41-48fd-9e75-78f4774c278b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:31:37.532815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.532776 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2458d771-9f41-48fd-9e75-78f4774c278b-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:37.532815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.532818 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2458d771-9f41-48fd-9e75-78f4774c278b-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:37.533038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:37.532834 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwdg5\" (UniqueName: \"kubernetes.io/projected/2458d771-9f41-48fd-9e75-78f4774c278b-kube-api-access-jwdg5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:38.187055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.187023 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" Oct 30 13:31:38.187580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.187064 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q" event={"ID":"2458d771-9f41-48fd-9e75-78f4774c278b","Type":"ContainerDied","Data":"0574da6312e4c7f6816d04e9acaddcad7217ba36ace31274c675b026d9900b90"} Oct 30 13:31:38.187580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.187125 5049 scope.go:122] "RemoveContainer" containerID="b5e38bee59081af1c7757b20e2e94cf7ad12cc4b04db92d681f69a1b85940c25" Oct 30 13:31:38.243026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.242989 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q"] Oct 30 13:31:38.250852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.250782 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/sub-test-model-ff4285-kserve-8965dffdc-lpg7q"] Oct 30 13:31:38.318345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.316901 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:38.636349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.633447 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:38.709343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.700484 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:38.788345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.788044 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-xks-ca\") pod \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " Oct 30 13:31:38.788345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.788217 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-kube-api-access-qgnxl\" (UniqueName: \"kubernetes.io/projected/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-kube-api-access-qgnxl\") pod \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " Oct 30 13:31:38.788622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.788416 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-openshift-ca\") pod \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\" (UID: \"d9f1b9ca-dcd0-411b-b2c3-cee56accd687\") " Oct 30 13:31:38.807347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.803997 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-openshift-ca" pod "d9f1b9ca-dcd0-411b-b2c3-cee56accd687" (UID: "d9f1b9ca-dcd0-411b-b2c3-cee56accd687"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:31:38.926395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:38.907730 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:39.137505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.137428 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58"] Oct 30 13:31:39.158402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.146132 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e57fb3-94t58"] Oct 30 13:31:39.159356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.158549 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-kube-api-access-qgnxl" pod "d9f1b9ca-dcd0-411b-b2c3-cee56accd687" (UID: "d9f1b9ca-dcd0-411b-b2c3-cee56accd687"). InnerVolumeSpecName "kube-api-access-qgnxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:31:39.236404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.222097 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgnxl\" (UniqueName: \"kubernetes.io/projected/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-kube-api-access-qgnxl\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:39.385359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.361021 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-xks-ca" pod "d9f1b9ca-dcd0-411b-b2c3-cee56accd687" (UID: "d9f1b9ca-dcd0-411b-b2c3-cee56accd687"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:31:39.468436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.443602 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f1b9ca-dcd0-411b-b2c3-cee56accd687-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:39.517181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.514146 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2458d771-9f41-48fd-9e75-78f4774c278b" path="/var/lib/kubelet/pods/2458d771-9f41-48fd-9e75-78f4774c278b/volumes" Oct 30 13:31:39.517181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.514922 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2978447-8e53-461f-9a9e-527f2dfc1452" path="/var/lib/kubelet/pods/a2978447-8e53-461f-9a9e-527f2dfc1452/volumes" Oct 30 13:31:39.588347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.573645 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" event={"ID":"d9f1b9ca-dcd0-411b-b2c3-cee56accd687","Type":"ContainerDied","Data":"c873a51a3cb547bc12d43f85e74d4a4929a6e115ee24357bed9ff8c54281cb03"} Oct 30 13:31:39.588347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.573713 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c873a51a3cb547bc12d43f85e74d4a4929a6e115ee24357bed9ff8c54281cb03" Oct 30 13:31:39.588347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.573796 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx" Oct 30 13:31:39.846397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:39.846230 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:40.369852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.369777 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:40.442506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.442471 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2"] Oct 30 13:31:40.443692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.443671 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d9f1b9ca-dcd0-411b-b2c3-cee56accd687" containerName="revoke-keys" Oct 30 13:31:40.443751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.443695 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f1b9ca-dcd0-411b-b2c3-cee56accd687" containerName="revoke-keys" Oct 30 13:31:40.443794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.443784 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2458d771-9f41-48fd-9e75-78f4774c278b" containerName="main" Oct 30 13:31:40.443834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.443793 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2458d771-9f41-48fd-9e75-78f4774c278b" containerName="main" Oct 30 13:31:40.444015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.443991 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2458d771-9f41-48fd-9e75-78f4774c278b" containerName="main" Oct 30 13:31:40.444015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.444019 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f1b9ca-dcd0-411b-b2c3-cee56accd687" containerName="revoke-keys" Oct 30 13:31:40.449939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.449916 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:31:40.464886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.464858 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6da60a\"" Oct 30 13:31:40.465135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.465059 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6da60a-dockercfg-crn4t\"" Oct 30 13:31:40.467999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.467975 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2"] Oct 30 13:31:40.521411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.518165 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48"] Oct 30 13:31:40.523719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.523694 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:31:40.547503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.547474 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:40.549574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.549548 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48"] Oct 30 13:31:40.597847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.576648 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05e65bd-2ee8-4b2e-9391-aa04ae888377-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2\" (UID: \"d05e65bd-2ee8-4b2e-9391-aa04ae888377\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:31:40.597847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.576802 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbhd8\" (UniqueName: \"kubernetes.io/projected/d05e65bd-2ee8-4b2e-9391-aa04ae888377-kube-api-access-bbhd8\") pod \"payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2\" (UID: \"d05e65bd-2ee8-4b2e-9391-aa04ae888377\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:31:40.691393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.684835 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snm8d\" (UniqueName: \"kubernetes.io/projected/f671b2e1-1907-476a-b1e8-6509e55ca6f6-kube-api-access-snm8d\") pod \"payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48\" (UID: \"f671b2e1-1907-476a-b1e8-6509e55ca6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:31:40.691393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.684919 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbhd8\" (UniqueName: \"kubernetes.io/projected/d05e65bd-2ee8-4b2e-9391-aa04ae888377-kube-api-access-bbhd8\") pod \"payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2\" (UID: \"d05e65bd-2ee8-4b2e-9391-aa04ae888377\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:31:40.691393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.685030 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f671b2e1-1907-476a-b1e8-6509e55ca6f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48\" (UID: \"f671b2e1-1907-476a-b1e8-6509e55ca6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:31:40.691393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.685260 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05e65bd-2ee8-4b2e-9391-aa04ae888377-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2\" (UID: \"d05e65bd-2ee8-4b2e-9391-aa04ae888377\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:31:40.691393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.686347 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05e65bd-2ee8-4b2e-9391-aa04ae888377-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2\" (UID: \"d05e65bd-2ee8-4b2e-9391-aa04ae888377\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:31:40.724572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.724443 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbhd8\" (UniqueName: \"kubernetes.io/projected/d05e65bd-2ee8-4b2e-9391-aa04ae888377-kube-api-access-bbhd8\") pod \"payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2\" (UID: \"d05e65bd-2ee8-4b2e-9391-aa04ae888377\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:31:40.769524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.769209 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:31:40.787909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.787866 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f671b2e1-1907-476a-b1e8-6509e55ca6f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48\" (UID: \"f671b2e1-1907-476a-b1e8-6509e55ca6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:31:40.788203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.788178 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snm8d\" (UniqueName: \"kubernetes.io/projected/f671b2e1-1907-476a-b1e8-6509e55ca6f6-kube-api-access-snm8d\") pod \"payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48\" (UID: \"f671b2e1-1907-476a-b1e8-6509e55ca6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:31:40.796069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.795750 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f671b2e1-1907-476a-b1e8-6509e55ca6f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48\" (UID: \"f671b2e1-1907-476a-b1e8-6509e55ca6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:31:40.856306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:40.856221 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snm8d\" (UniqueName: \"kubernetes.io/projected/f671b2e1-1907-476a-b1e8-6509e55ca6f6-kube-api-access-snm8d\") pod \"payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48\" (UID: \"f671b2e1-1907-476a-b1e8-6509e55ca6f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:31:41.111929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.111899 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2"] Oct 30 13:31:41.115427 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:31:41.115399 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05e65bd_2ee8_4b2e_9391_aa04ae888377.slice/crio-be3d580e01ade1ed7d47b6c816073026232c06912f8ce9c7f91096d3ff30f82b WatchSource:0}: Error finding container be3d580e01ade1ed7d47b6c816073026232c06912f8ce9c7f91096d3ff30f82b: Status 404 returned error can't find the container with id be3d580e01ade1ed7d47b6c816073026232c06912f8ce9c7f91096d3ff30f82b Oct 30 13:31:41.140750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.140717 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:31:41.476185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.476157 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48"] Oct 30 13:31:41.479084 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:31:41.479056 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf671b2e1_1907_476a_b1e8_6509e55ca6f6.slice/crio-9897b2d90f9f75d1ffd2de5c876ec3626766670584e6f74968e28f0be2a540cb WatchSource:0}: Error finding container 9897b2d90f9f75d1ffd2de5c876ec3626766670584e6f74968e28f0be2a540cb: Status 404 returned error can't find the container with id 9897b2d90f9f75d1ffd2de5c876ec3626766670584e6f74968e28f0be2a540cb Oct 30 13:31:41.734389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.733952 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" event={"ID":"f671b2e1-1907-476a-b1e8-6509e55ca6f6","Type":"ContainerStarted","Data":"139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13"} Oct 30 13:31:41.734389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.734016 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" event={"ID":"f671b2e1-1907-476a-b1e8-6509e55ca6f6","Type":"ContainerStarted","Data":"9897b2d90f9f75d1ffd2de5c876ec3626766670584e6f74968e28f0be2a540cb"} Oct 30 13:31:41.734389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.734048 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:31:41.735806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.735780 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" event={"ID":"d05e65bd-2ee8-4b2e-9391-aa04ae888377","Type":"ContainerStarted","Data":"83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635"} Oct 30 13:31:41.735961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.735827 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" event={"ID":"d05e65bd-2ee8-4b2e-9391-aa04ae888377","Type":"ContainerStarted","Data":"be3d580e01ade1ed7d47b6c816073026232c06912f8ce9c7f91096d3ff30f82b"} Oct 30 13:31:41.735961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.735869 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:31:41.775419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.775337 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" podStartSLOduration=1.775303055 podStartE2EDuration="1.775303055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:41.772455675 +0000 UTC m=+2408.587650099" watchObservedRunningTime="2026-10-30 13:31:41.775303055 +0000 UTC m=+2408.590497479" Oct 30 13:31:41.814508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:41.814445 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" podStartSLOduration=1.814425629 podStartE2EDuration="1.814425629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:41.811688686 +0000 UTC m=+2408.626883112" watchObservedRunningTime="2026-10-30 13:31:41.814425629 +0000 UTC m=+2408.629620053" Oct 30 13:31:42.154945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:42.154900 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:44.125345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:44.125137 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:45.344345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.344172 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn"] Oct 30 13:31:45.357994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.356344 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:45.360285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.360252 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd59d610-f816-43e2-8d11-59286b028201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d6ce356-24gfn\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:45.360519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.360500 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd59d610-f816-43e2-8d11-59286b028201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d6ce356-24gfn\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:45.360801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.360774 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvcq\" (UniqueName: \"kubernetes.io/projected/dd59d610-f816-43e2-8d11-59286b028201-kube-api-access-4xvcq\") pod \"maas-api-revoke-sub-7d6ce356-24gfn\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:45.368916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.368888 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn"] Oct 30 13:31:45.372447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.370118 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:31:45.379825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.379805 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:31:45.461570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.461536 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd59d610-f816-43e2-8d11-59286b028201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d6ce356-24gfn\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:45.461750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.461654 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd59d610-f816-43e2-8d11-59286b028201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d6ce356-24gfn\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:45.461818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.461801 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvcq\" (UniqueName: \"kubernetes.io/projected/dd59d610-f816-43e2-8d11-59286b028201-kube-api-access-4xvcq\") pod \"maas-api-revoke-sub-7d6ce356-24gfn\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:45.462292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.462261 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd59d610-f816-43e2-8d11-59286b028201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d6ce356-24gfn\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:45.463983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.463954 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd59d610-f816-43e2-8d11-59286b028201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d6ce356-24gfn\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:45.483498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.483410 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvcq\" (UniqueName: \"kubernetes.io/projected/dd59d610-f816-43e2-8d11-59286b028201-kube-api-access-4xvcq\") pod \"maas-api-revoke-sub-7d6ce356-24gfn\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:45.683807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:45.683727 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:46.004011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:46.003982 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn"] Oct 30 13:31:46.007335 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:31:46.007287 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd59d610_f816_43e2_8d11_59286b028201.slice/crio-4fc8223b2438450182a4b9792b554acfab782529124e25d8bce0fc720d325398 WatchSource:0}: Error finding container 4fc8223b2438450182a4b9792b554acfab782529124e25d8bce0fc720d325398: Status 404 returned error can't find the container with id 4fc8223b2438450182a4b9792b554acfab782529124e25d8bce0fc720d325398 Oct 30 13:31:46.140346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:46.139873 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:46.803985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:46.803953 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" event={"ID":"dd59d610-f816-43e2-8d11-59286b028201","Type":"ContainerStarted","Data":"ef2234e12655d5feaff38e75ee909a44076b87a25e6a2bccf71481d6001abb1c"} Oct 30 13:31:46.804402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:46.804013 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" event={"ID":"dd59d610-f816-43e2-8d11-59286b028201","Type":"ContainerStarted","Data":"4fc8223b2438450182a4b9792b554acfab782529124e25d8bce0fc720d325398"} Oct 30 13:31:46.855015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:46.854955 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" podStartSLOduration=1.854939406 podStartE2EDuration="1.854939406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:46.841949718 +0000 UTC m=+2413.657144141" watchObservedRunningTime="2026-10-30 13:31:46.854939406 +0000 UTC m=+2413.670133829" Oct 30 13:31:47.818455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:47.818419 5049 generic.go:356] "Generic (PLEG): container finished" podID="dd59d610-f816-43e2-8d11-59286b028201" containerID="ef2234e12655d5feaff38e75ee909a44076b87a25e6a2bccf71481d6001abb1c" exitCode=0 Oct 30 13:31:47.818875 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:47.818508 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" event={"ID":"dd59d610-f816-43e2-8d11-59286b028201","Type":"ContainerDied","Data":"ef2234e12655d5feaff38e75ee909a44076b87a25e6a2bccf71481d6001abb1c"} Oct 30 13:31:48.006195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:48.006152 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:49.024805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.024775 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:49.121906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.121807 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd59d610-f816-43e2-8d11-59286b028201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd59d610-f816-43e2-8d11-59286b028201-maas-api-xks-ca\") pod \"dd59d610-f816-43e2-8d11-59286b028201\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " Oct 30 13:31:49.122289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.122260 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd59d610-f816-43e2-8d11-59286b028201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd59d610-f816-43e2-8d11-59286b028201-maas-api-openshift-ca\") pod \"dd59d610-f816-43e2-8d11-59286b028201\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " Oct 30 13:31:49.122531 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.122505 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd59d610-f816-43e2-8d11-59286b028201-kube-api-access-4xvcq\" (UniqueName: \"kubernetes.io/projected/dd59d610-f816-43e2-8d11-59286b028201-kube-api-access-4xvcq\") pod \"dd59d610-f816-43e2-8d11-59286b028201\" (UID: \"dd59d610-f816-43e2-8d11-59286b028201\") " Oct 30 13:31:49.122743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.122713 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd59d610-f816-43e2-8d11-59286b028201-maas-api-openshift-ca" pod "dd59d610-f816-43e2-8d11-59286b028201" (UID: "dd59d610-f816-43e2-8d11-59286b028201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:31:49.123270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.123241 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd59d610-f816-43e2-8d11-59286b028201-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:49.124588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.124561 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd59d610-f816-43e2-8d11-59286b028201-maas-api-xks-ca" pod "dd59d610-f816-43e2-8d11-59286b028201" (UID: "dd59d610-f816-43e2-8d11-59286b028201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:31:49.125748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.125719 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd59d610-f816-43e2-8d11-59286b028201-kube-api-access-4xvcq" pod "dd59d610-f816-43e2-8d11-59286b028201" (UID: "dd59d610-f816-43e2-8d11-59286b028201"). InnerVolumeSpecName "kube-api-access-4xvcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:31:49.225568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.225530 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xvcq\" (UniqueName: \"kubernetes.io/projected/dd59d610-f816-43e2-8d11-59286b028201-kube-api-access-4xvcq\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:49.225568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.225561 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd59d610-f816-43e2-8d11-59286b028201-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:31:49.840524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.840490 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" Oct 30 13:31:49.840702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.840527 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn" event={"ID":"dd59d610-f816-43e2-8d11-59286b028201","Type":"ContainerDied","Data":"4fc8223b2438450182a4b9792b554acfab782529124e25d8bce0fc720d325398"} Oct 30 13:31:49.840702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:49.840577 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fc8223b2438450182a4b9792b554acfab782529124e25d8bce0fc720d325398" Oct 30 13:31:50.007017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:50.006980 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:51.899091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:51.899045 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:52.148431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:52.148395 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:52.752595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:52.752563 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:31:52.752754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:52.752653 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:31:53.799305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:53.799259 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:56.396532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:56.396496 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:56.680810 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:31:56.680734 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:31:58.081279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:58.081234 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:59.863727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:31:59.863677 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:01.740517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:01.740470 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:01.924869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:01.924837 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:03.578904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:03.578871 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:05.450758 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:05.450715 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:05.618603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:05.618565 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:07.609035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:07.608970 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:09.741732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:09.741699 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:11.616862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:11.616805 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:13.596554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:13.596523 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:14.488532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.488495 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7"] Oct 30 13:32:14.489307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.489291 5049 cpu_manager.go:414] "containerMap: removing container" podUID="dd59d610-f816-43e2-8d11-59286b028201" containerName="revoke-keys" Oct 30 13:32:14.489419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.489309 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd59d610-f816-43e2-8d11-59286b028201" containerName="revoke-keys" Oct 30 13:32:14.489498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.489465 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd59d610-f816-43e2-8d11-59286b028201" containerName="revoke-keys" Oct 30 13:32:14.494888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.494867 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.519920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.519798 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"default-dockercfg-565c8\"" Oct 30 13:32:14.519920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.519880 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"rate-test-model-ff4285-kserve-self-signed-certs\"" Oct 30 13:32:14.521349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.521294 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"openshift-service-ca.crt\"" Oct 30 13:32:14.521470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.521370 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"kube-root-ca.crt\"" Oct 30 13:32:14.534245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.534199 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7"] Oct 30 13:32:14.598788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.598756 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:14.638546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.638513 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-home\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.638732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.638575 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdjl\" (UniqueName: \"kubernetes.io/projected/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-kube-api-access-nwdjl\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.638732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.638611 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-model-cache\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.638811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.638718 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tls-certs\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.638916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.638888 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tmp-dir\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.639044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.638976 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-dshm\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.741065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.740973 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tmp-dir\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.741065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.741048 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-dshm\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.741298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.741273 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-home\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.741405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.741384 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdjl\" (UniqueName: \"kubernetes.io/projected/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-kube-api-access-nwdjl\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.741477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.741410 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tmp-dir\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.741477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.741426 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-model-cache\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.741593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.741480 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tls-certs\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.741593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.741578 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-home\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.741739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.741719 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-model-cache\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.743840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.743811 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-dshm\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.744060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.744038 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tls-certs\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.790053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.790021 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdjl\" (UniqueName: \"kubernetes.io/projected/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-kube-api-access-nwdjl\") pod \"rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:14.815558 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:14.815228 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:15.177999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:15.177970 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7"] Oct 30 13:32:15.181211 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:32:15.181183 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68fc0e80_aed3_4ebc_8a2e_78ffba42e75f.slice/crio-0e6ac932db1f02c6a8eae618f6ba9377232701d3eb98e3d124ca50e878b4d85b WatchSource:0}: Error finding container 0e6ac932db1f02c6a8eae618f6ba9377232701d3eb98e3d124ca50e878b4d85b: Status 404 returned error can't find the container with id 0e6ac932db1f02c6a8eae618f6ba9377232701d3eb98e3d124ca50e878b4d85b Oct 30 13:32:15.430592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:15.430444 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:16.087104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.087069 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" event={"ID":"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f","Type":"ContainerStarted","Data":"24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2"} Oct 30 13:32:16.087558 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.087114 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" event={"ID":"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f","Type":"ContainerStarted","Data":"0e6ac932db1f02c6a8eae618f6ba9377232701d3eb98e3d124ca50e878b4d85b"} Oct 30 13:32:16.135038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.134972 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" podStartSLOduration=2.134952071 podStartE2EDuration="2.134952071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:16.130335317 +0000 UTC m=+2442.945529739" watchObservedRunningTime="2026-10-30 13:32:16.134952071 +0000 UTC m=+2442.950146494" Oct 30 13:32:16.817009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.816973 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:16.922539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.922507 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr"] Oct 30 13:32:16.934716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.934599 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:16.962607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.962577 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-9cbcbc\"/\"openshift-service-ca.crt\"" Oct 30 13:32:16.962784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.962685 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-9cbcbc\"/\"auth-test-model-9cbcbc-kserve-self-signed-certs\"" Oct 30 13:32:16.964705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.964676 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-9cbcbc\"/\"default-dockercfg-cv2s8\"" Oct 30 13:32:16.964823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.964718 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-9cbcbc\"/\"kube-root-ca.crt\"" Oct 30 13:32:16.970704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:16.970659 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr"] Oct 30 13:32:17.079709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.079624 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:17.083183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.083150 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qm87\" (UniqueName: \"kubernetes.io/projected/17fd01c8-8e29-445d-be8f-156b8159f469-kube-api-access-4qm87\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.083377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.083210 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-tmp-dir\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.083456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.083441 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-dshm\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.083520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.083477 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-model-cache\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.083586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.083559 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-home\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.083664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.083648 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17fd01c8-8e29-445d-be8f-156b8159f469-tls-certs\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.184987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.184949 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-dshm\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.185506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.185017 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-model-cache\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.185506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.185051 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-home\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.185506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.185114 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17fd01c8-8e29-445d-be8f-156b8159f469-tls-certs\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.185506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.185262 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qm87\" (UniqueName: \"kubernetes.io/projected/17fd01c8-8e29-445d-be8f-156b8159f469-kube-api-access-4qm87\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.185506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.185352 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-tmp-dir\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.185506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.185493 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-model-cache\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.185832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.185761 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-tmp-dir\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.186007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.185988 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-home\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.201370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.189031 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17fd01c8-8e29-445d-be8f-156b8159f469-tls-certs\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.201370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.189250 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-dshm\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.231529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.231491 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qm87\" (UniqueName: \"kubernetes.io/projected/17fd01c8-8e29-445d-be8f-156b8159f469-kube-api-access-4qm87\") pod \"auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.261986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.261954 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:17.509474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.507756 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:17.647733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:17.647683 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr"] Oct 30 13:32:17.649396 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:32:17.649303 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17fd01c8_8e29_445d_be8f_156b8159f469.slice/crio-62104196cb37b2ad116b152ee7444957fe4eefe2482b8f0c9cf02a196387848f WatchSource:0}: Error finding container 62104196cb37b2ad116b152ee7444957fe4eefe2482b8f0c9cf02a196387848f: Status 404 returned error can't find the container with id 62104196cb37b2ad116b152ee7444957fe4eefe2482b8f0c9cf02a196387848f Oct 30 13:32:18.109966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:18.109922 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" event={"ID":"17fd01c8-8e29-445d-be8f-156b8159f469","Type":"ContainerStarted","Data":"40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1"} Oct 30 13:32:18.110200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:18.109981 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" event={"ID":"17fd01c8-8e29-445d-be8f-156b8159f469","Type":"ContainerStarted","Data":"62104196cb37b2ad116b152ee7444957fe4eefe2482b8f0c9cf02a196387848f"} Oct 30 13:32:18.158024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:18.157958 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" podStartSLOduration=2.15793895 podStartE2EDuration="2.15793895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:18.154186017 +0000 UTC m=+2444.969380443" watchObservedRunningTime="2026-10-30 13:32:18.15793895 +0000 UTC m=+2444.973133375" Oct 30 13:32:18.263384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:18.263349 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:18.997981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:18.994798 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:19.586434 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:19.586395 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:21.349552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:21.349513 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:21.530431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:21.530393 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:21.699985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:21.699907 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:23.514361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:23.513901 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:24.816521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:24.816481 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:24.830532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:24.830499 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:25.274561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:25.274474 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:32:25.743849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:25.743814 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:26.068695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:26.068665 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:26.679299 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:32:26.679263 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:32:27.263308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:27.263271 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:27.276626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:27.276598 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:27.290546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:27.290522 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:32:27.653555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:27.653504 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:28.855033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:28.854998 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:29.521120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:29.521028 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:30.929217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:30.929178 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:31.508000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:31.507964 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:33.324226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:33.324186 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:35.219664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:35.219630 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:36.264950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:36.264914 5049 scope.go:122] "RemoveContainer" containerID="605274dc121be8585ac4d1e64b851d4418bcc1b5c5c296fe2dc7b0719aa7feef" Oct 30 13:32:36.276552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:36.276532 5049 scope.go:122] "RemoveContainer" containerID="8718c51224b2c5afdd5ae3c6c61be77cec3a7a9b2ea3926cc7f897cda504e7f6" Oct 30 13:32:36.287183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:36.287160 5049 scope.go:122] "RemoveContainer" containerID="b8c22acf5be1117e4ceef176421f8f260543df0e24f21c4938b48b5c07f2855e" Oct 30 13:32:36.297370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:36.297350 5049 scope.go:122] "RemoveContainer" containerID="daa9ae4250e45dd9c80a5b873f18391ef8848ab4f155744c3fc19cded5a6cb99" Oct 30 13:32:36.308419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:36.308392 5049 scope.go:122] "RemoveContainer" containerID="d80d9494773d9cffac53fa507d536dfdada8dae9bfaa4ad70a8e564b1d6b2440" Oct 30 13:32:36.319029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:36.319003 5049 scope.go:122] "RemoveContainer" containerID="fcf84099b0c55dad51818b0fc7838729761b5b1da69ccb6383edf856f3ab951b" Oct 30 13:32:37.019041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:37.019004 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:38.874170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:38.874079 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:40.439849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:40.439801 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:40.925713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:40.925670 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:41.167990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:41.167950 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:41.623789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:41.623755 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:41.829346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:41.819969 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:42.785360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:42.785044 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:44.566398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.566279 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w"] Oct 30 13:32:44.573436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.573412 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.587231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.587210 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4b0ea9-openshift-default-dockercfg-wbmk9\"" Oct 30 13:32:44.602565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.602539 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w"] Oct 30 13:32:44.649177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.649140 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-credential-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.649412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.649200 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-data\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.649412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.649338 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8111c92e-8da9-49eb-ac5f-14427435e155-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.649412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.649374 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.649628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.649499 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwq8\" (UniqueName: \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-kube-api-access-9rwq8\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.649628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.649521 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8111c92e-8da9-49eb-ac5f-14427435e155-istio-podinfo\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.649628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.649567 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-certs\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.649871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.649691 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-istio-token\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.649871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.649736 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-envoy\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.751576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.751540 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwq8\" (UniqueName: \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-kube-api-access-9rwq8\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.751576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.751582 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8111c92e-8da9-49eb-ac5f-14427435e155-istio-podinfo\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.751847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.751652 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-certs\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.751847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.751721 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-istio-token\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.751847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.751777 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-envoy\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.751847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.751830 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-credential-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.752081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.751885 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-data\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.752081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.751944 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8111c92e-8da9-49eb-ac5f-14427435e155-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.752081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.751986 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.752225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.752184 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-certs\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.752304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.752280 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-credential-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.752458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.752394 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-data\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.752458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.752402 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.752928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.752907 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8111c92e-8da9-49eb-ac5f-14427435e155-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.754299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.754277 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-envoy\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.754548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.754525 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8111c92e-8da9-49eb-ac5f-14427435e155-istio-podinfo\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.774071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.774040 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-istio-token\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.774365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.774345 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwq8\" (UniqueName: \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-kube-api-access-9rwq8\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:44.817229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.816134 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:44.888761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:44.888729 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:45.243075 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:32:45.243044 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8111c92e_8da9_49eb_ac5f_14427435e155.slice/crio-e2645ad10e55145033fc7d4446da30080f50751d0f8d4a6c631c24339cd30931 WatchSource:0}: Error finding container e2645ad10e55145033fc7d4446da30080f50751d0f8d4a6c631c24339cd30931: Status 404 returned error can't find the container with id e2645ad10e55145033fc7d4446da30080f50751d0f8d4a6c631c24339cd30931 Oct 30 13:32:45.243903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:45.243875 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w"] Oct 30 13:32:45.246095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:45.246064 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:45.246185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:45.246130 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:45.246185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:45.246159 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:45.468419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:45.468380 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" event={"ID":"8111c92e-8da9-49eb-ac5f-14427435e155","Type":"ContainerStarted","Data":"4d2c5f4703de51275fdb6aa5e549b87e61acd70e0ac016bf208674524b3ceb0a"} Oct 30 13:32:45.468632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:45.468441 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" event={"ID":"8111c92e-8da9-49eb-ac5f-14427435e155","Type":"ContainerStarted","Data":"e2645ad10e55145033fc7d4446da30080f50751d0f8d4a6c631c24339cd30931"} Oct 30 13:32:45.510173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:45.510081 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:45.519203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:45.519145 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" podStartSLOduration=1.519131252 podStartE2EDuration="1.519131252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:45.512764028 +0000 UTC m=+2472.327958451" watchObservedRunningTime="2026-10-30 13:32:45.519131252 +0000 UTC m=+2472.334325669" Oct 30 13:32:45.681837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:45.681781 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:45.889184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:45.889134 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:46.890418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:46.890383 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 13:32:46.890828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:46.890442 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" podUID="8111c92e-8da9-49eb-ac5f-14427435e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 13:32:47.177107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:47.177006 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:47.452484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:47.452389 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:47.700270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:47.700228 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:47.901977 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:47.901940 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 13:32:47.902597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:47.902565 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" podUID="8111c92e-8da9-49eb-ac5f-14427435e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 13:32:48.889744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:48.889707 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 13:32:48.889973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:48.889764 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" podUID="8111c92e-8da9-49eb-ac5f-14427435e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 13:32:49.144691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:49.144603 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:49.889852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:49.889817 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 13:32:49.890050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:49.889881 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" podUID="8111c92e-8da9-49eb-ac5f-14427435e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 13:32:50.040563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:50.040523 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg"] Oct 30 13:32:50.047870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:50.047838 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63997418-gz4zg"] Oct 30 13:32:50.890165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:50.890127 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 13:32:50.890605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:50.890197 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" podUID="8111c92e-8da9-49eb-ac5f-14427435e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 13:32:50.893854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:50.893827 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:51.343253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:51.343222 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08599d79-4e93-4a7f-8d30-d4034ab55db8" path="/var/lib/kubelet/pods/08599d79-4e93-4a7f-8d30-d4034ab55db8/volumes" Oct 30 13:32:52.008609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:52.008548 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:52.010588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:52.010533 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:52.013143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:52.013117 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:32:52.762193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:52.762145 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:54.496031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:54.494284 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:56.291526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:56.291489 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:56.679314 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:32:56.679275 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:32:57.221028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:57.220992 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:58.184895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:32:58.184862 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:00.204583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:00.204545 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:01.907212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:01.906794 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:03.859985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:03.859947 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:04.285590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:04.285503 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:05.882162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:05.882014 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:07.973785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:07.973658 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:08.487661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:08.487630 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:10.109345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:10.086402 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:10.154346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:10.153644 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs"] Oct 30 13:33:10.190345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:10.189541 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6e4f8f-m65bs"] Oct 30 13:33:10.665313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:10.665281 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:11.343485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:11.343443 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfb32b0-b999-4f64-ba2f-efa5eb135bd4" path="/var/lib/kubelet/pods/8cfb32b0-b999-4f64-ba2f-efa5eb135bd4/volumes" Oct 30 13:33:12.313675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:12.313638 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:14.184116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:14.184059 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:15.975356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:15.975304 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:17.928883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:17.928838 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:19.763014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:19.762981 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:21.902410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:21.902374 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:23.738726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:23.738683 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:25.563226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:25.563189 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:26.678473 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:33:26.678437 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:33:27.379642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:27.379599 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:27.544077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:27.544043 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:27.718869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:27.718786 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:28.211883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.211846 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb"] Oct 30 13:33:28.218996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.218972 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.241945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.241916 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-fdd4d4\"/\"openshift-service-ca.crt\"" Oct 30 13:33:28.242127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.241992 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-fdd4d4\"/\"default-dockercfg-hckq2\"" Oct 30 13:33:28.242369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.242350 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-fdd4d4\"/\"auth-test-model-fdd4d4-kserve-self-signed-certs\"" Oct 30 13:33:28.244659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.244635 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-fdd4d4\"/\"kube-root-ca.crt\"" Oct 30 13:33:28.258903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.258868 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb"] Oct 30 13:33:28.309157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.309121 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-home\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.309377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.309270 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-dshm\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.309454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.309380 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56dk4\" (UniqueName: \"kubernetes.io/projected/9e2b7bd0-8626-4025-bc7e-546b26a77764-kube-api-access-56dk4\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.309542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.309519 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2b7bd0-8626-4025-bc7e-546b26a77764-tls-certs\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.309601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.309583 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-model-cache\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.309776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.309756 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-tmp-dir\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.350219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.350193 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:28.411606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.411573 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-dshm\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.411780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.411665 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56dk4\" (UniqueName: \"kubernetes.io/projected/9e2b7bd0-8626-4025-bc7e-546b26a77764-kube-api-access-56dk4\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.411780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.411744 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2b7bd0-8626-4025-bc7e-546b26a77764-tls-certs\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.411938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.411790 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-model-cache\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.412007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.411938 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-tmp-dir\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.412106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.412085 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-home\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.412299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.412274 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-model-cache\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.412752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.412733 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-tmp-dir\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.412897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.412852 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-home\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.414211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.414188 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-dshm\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.414685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.414666 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2b7bd0-8626-4025-bc7e-546b26a77764-tls-certs\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.441353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.441307 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56dk4\" (UniqueName: \"kubernetes.io/projected/9e2b7bd0-8626-4025-bc7e-546b26a77764-kube-api-access-56dk4\") pod \"auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.537921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.537831 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:28.894274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.894247 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb"] Oct 30 13:33:28.896100 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:33:28.896069 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e2b7bd0_8626_4025_bc7e_546b26a77764.slice/crio-7fdc97c9f1e221e7ee6f27164750082ae6c29edc2442708c0616211c403e8176 WatchSource:0}: Error finding container 7fdc97c9f1e221e7ee6f27164750082ae6c29edc2442708c0616211c403e8176: Status 404 returned error can't find the container with id 7fdc97c9f1e221e7ee6f27164750082ae6c29edc2442708c0616211c403e8176 Oct 30 13:33:28.984096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:28.984066 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:29.119712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:29.119674 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" event={"ID":"9e2b7bd0-8626-4025-bc7e-546b26a77764","Type":"ContainerStarted","Data":"0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a"} Oct 30 13:33:29.119911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:29.119722 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" event={"ID":"9e2b7bd0-8626-4025-bc7e-546b26a77764","Type":"ContainerStarted","Data":"7fdc97c9f1e221e7ee6f27164750082ae6c29edc2442708c0616211c403e8176"} Oct 30 13:33:29.165786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:29.165692 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" podStartSLOduration=1.165678916 podStartE2EDuration="1.165678916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:29.161264606 +0000 UTC m=+2515.976459029" watchObservedRunningTime="2026-10-30 13:33:29.165678916 +0000 UTC m=+2515.980873366" Oct 30 13:33:29.500109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:29.500024 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:29.538383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:29.538340 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:31.319005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:31.318933 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:33.142807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:33.142774 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:33.775469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:33.775431 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:35.062356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:35.062296 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:36.390165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:36.390133 5049 scope.go:122] "RemoveContainer" containerID="6dce3bc59ff35fbf1bc125643ff988f26ec45bec83e179c377c53c9c08c14617" Oct 30 13:33:36.401998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:36.401961 5049 scope.go:122] "RemoveContainer" containerID="ee57f56b9634c88d16927b16de118710c16f9e37db6030c845dce8613de3e758" Oct 30 13:33:36.957932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:36.957898 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:38.538853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:38.538815 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:38.558431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:38.558399 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:38.799583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:38.799542 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:39.325343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:39.317661 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:33:41.134002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:41.133966 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:43.229063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:43.229001 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:44.375580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.375544 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr"] Oct 30 13:33:44.380433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.380411 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:44.402443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.402405 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr"] Oct 30 13:33:44.405764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.405735 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:33:44.405915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.405769 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:33:44.466783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.466741 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07129f75-fdpkr\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:44.467016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.466857 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07129f75-fdpkr\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:44.467016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.466961 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdxl\" (UniqueName: \"kubernetes.io/projected/9ad73dbe-40ec-4e07-a567-0d3b811063b6-kube-api-access-sfdxl\") pod \"maas-api-revoke-sub-07129f75-fdpkr\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:44.568956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.568917 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07129f75-fdpkr\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:44.569146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.568971 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07129f75-fdpkr\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:44.569146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.569034 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdxl\" (UniqueName: \"kubernetes.io/projected/9ad73dbe-40ec-4e07-a567-0d3b811063b6-kube-api-access-sfdxl\") pod \"maas-api-revoke-sub-07129f75-fdpkr\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:44.569671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.569646 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07129f75-fdpkr\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:44.571419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.571392 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07129f75-fdpkr\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:44.590633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.590611 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdxl\" (UniqueName: \"kubernetes.io/projected/9ad73dbe-40ec-4e07-a567-0d3b811063b6-kube-api-access-sfdxl\") pod \"maas-api-revoke-sub-07129f75-fdpkr\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:44.695803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:44.695722 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:45.015284 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:33:45.015250 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad73dbe_40ec_4e07_a567_0d3b811063b6.slice/crio-11bac4938abcfc8513a69864fc5ad9b2682ec44eafdd5460a03806fe0ea6a38c WatchSource:0}: Error finding container 11bac4938abcfc8513a69864fc5ad9b2682ec44eafdd5460a03806fe0ea6a38c: Status 404 returned error can't find the container with id 11bac4938abcfc8513a69864fc5ad9b2682ec44eafdd5460a03806fe0ea6a38c Oct 30 13:33:45.015486 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:45.015407 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr"] Oct 30 13:33:45.157037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:45.156999 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:45.343819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:45.343778 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" event={"ID":"9ad73dbe-40ec-4e07-a567-0d3b811063b6","Type":"ContainerStarted","Data":"11bac4938abcfc8513a69864fc5ad9b2682ec44eafdd5460a03806fe0ea6a38c"} Oct 30 13:33:46.257969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.257931 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj"] Oct 30 13:33:46.262839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.262804 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.285253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.285117 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj"] Oct 30 13:33:46.357648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.357610 5049 generic.go:356] "Generic (PLEG): container finished" podID="9ad73dbe-40ec-4e07-a567-0d3b811063b6" containerID="00ced1778ea658c177bac557254551617094586f3a41a0c8457a32dd41c09c7b" exitCode=0 Oct 30 13:33:46.357868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.357691 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" event={"ID":"9ad73dbe-40ec-4e07-a567-0d3b811063b6","Type":"ContainerDied","Data":"00ced1778ea658c177bac557254551617094586f3a41a0c8457a32dd41c09c7b"} Oct 30 13:33:46.400562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.400513 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c279e4-7510-4613-865d-88b96986a27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ebfa0e-4wzxj\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.400760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.400699 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c279e4-7510-4613-865d-88b96986a27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ebfa0e-4wzxj\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.400760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.400733 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8w6l\" (UniqueName: \"kubernetes.io/projected/99c279e4-7510-4613-865d-88b96986a27c-kube-api-access-w8w6l\") pod \"maas-api-revoke-sub-c8ebfa0e-4wzxj\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.502312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.502274 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c279e4-7510-4613-865d-88b96986a27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ebfa0e-4wzxj\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.502548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.502432 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c279e4-7510-4613-865d-88b96986a27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ebfa0e-4wzxj\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.502548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.502469 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8w6l\" (UniqueName: \"kubernetes.io/projected/99c279e4-7510-4613-865d-88b96986a27c-kube-api-access-w8w6l\") pod \"maas-api-revoke-sub-c8ebfa0e-4wzxj\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.503155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.503130 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c279e4-7510-4613-865d-88b96986a27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ebfa0e-4wzxj\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.504852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.504826 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c279e4-7510-4613-865d-88b96986a27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ebfa0e-4wzxj\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.526019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.525991 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz"] Oct 30 13:33:46.526218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.526195 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8w6l\" (UniqueName: \"kubernetes.io/projected/99c279e4-7510-4613-865d-88b96986a27c-kube-api-access-w8w6l\") pod \"maas-api-revoke-sub-c8ebfa0e-4wzxj\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.532416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.532393 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.556170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.556139 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz"] Oct 30 13:33:46.579261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.579232 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:46.605161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.604457 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdbd\" (UniqueName: \"kubernetes.io/projected/d981c28b-a864-4825-841f-df7978785cc8-kube-api-access-txdbd\") pod \"maas-api-revoke-sub-30230832-pp4fz\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.605161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.604739 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d981c28b-a864-4825-841f-df7978785cc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30230832-pp4fz\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.605161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.604873 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d981c28b-a864-4825-841f-df7978785cc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30230832-pp4fz\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.706510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.706472 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d981c28b-a864-4825-841f-df7978785cc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30230832-pp4fz\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.706707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.706603 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d981c28b-a864-4825-841f-df7978785cc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30230832-pp4fz\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.706781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.706751 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdbd\" (UniqueName: \"kubernetes.io/projected/d981c28b-a864-4825-841f-df7978785cc8-kube-api-access-txdbd\") pod \"maas-api-revoke-sub-30230832-pp4fz\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.707267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.707238 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d981c28b-a864-4825-841f-df7978785cc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30230832-pp4fz\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.714362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.711645 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d981c28b-a864-4825-841f-df7978785cc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30230832-pp4fz\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.740468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.734595 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdbd\" (UniqueName: \"kubernetes.io/projected/d981c28b-a864-4825-841f-df7978785cc8-kube-api-access-txdbd\") pod \"maas-api-revoke-sub-30230832-pp4fz\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.866553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.866475 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:46.946748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:46.944388 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj"] Oct 30 13:33:47.196189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.196161 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz"] Oct 30 13:33:47.210249 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:33:47.210216 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd981c28b_a864_4825_841f_df7978785cc8.slice/crio-60b14c30a1a727339241e871a82cecedf284b82ff1b757a2d7fcd1c41b211397 WatchSource:0}: Error finding container 60b14c30a1a727339241e871a82cecedf284b82ff1b757a2d7fcd1c41b211397: Status 404 returned error can't find the container with id 60b14c30a1a727339241e871a82cecedf284b82ff1b757a2d7fcd1c41b211397 Oct 30 13:33:47.218430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.218405 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:47.368210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.368174 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" event={"ID":"99c279e4-7510-4613-865d-88b96986a27c","Type":"ContainerStarted","Data":"a3424188960fb88271c2fc60d8293df5974ea6d91dbe73d70b0ff8c4c5e09843"} Oct 30 13:33:47.369723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.369695 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" event={"ID":"d981c28b-a864-4825-841f-df7978785cc8","Type":"ContainerStarted","Data":"60b14c30a1a727339241e871a82cecedf284b82ff1b757a2d7fcd1c41b211397"} Oct 30 13:33:47.574858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.574824 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:47.727548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.727515 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-xks-ca\") pod \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " Oct 30 13:33:47.746716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.728089 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-openshift-ca\") pod \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " Oct 30 13:33:47.746716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.728490 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad73dbe-40ec-4e07-a567-0d3b811063b6-kube-api-access-sfdxl\" (UniqueName: \"kubernetes.io/projected/9ad73dbe-40ec-4e07-a567-0d3b811063b6-kube-api-access-sfdxl\") pod \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\" (UID: \"9ad73dbe-40ec-4e07-a567-0d3b811063b6\") " Oct 30 13:33:47.746716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.728581 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-openshift-ca" pod "9ad73dbe-40ec-4e07-a567-0d3b811063b6" (UID: "9ad73dbe-40ec-4e07-a567-0d3b811063b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:47.749372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.747148 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad73dbe-40ec-4e07-a567-0d3b811063b6-kube-api-access-sfdxl" pod "9ad73dbe-40ec-4e07-a567-0d3b811063b6" (UID: "9ad73dbe-40ec-4e07-a567-0d3b811063b6"). InnerVolumeSpecName "kube-api-access-sfdxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:47.749372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.748687 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-xks-ca" pod "9ad73dbe-40ec-4e07-a567-0d3b811063b6" (UID: "9ad73dbe-40ec-4e07-a567-0d3b811063b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:33:47.749649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.749624 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfdxl\" (UniqueName: \"kubernetes.io/projected/9ad73dbe-40ec-4e07-a567-0d3b811063b6-kube-api-access-sfdxl\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:47.749704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.749652 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:47.749704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:47.749664 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ad73dbe-40ec-4e07-a567-0d3b811063b6-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:48.380201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:48.380154 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" event={"ID":"d981c28b-a864-4825-841f-df7978785cc8","Type":"ContainerStarted","Data":"f67687d9a123a0f33ef133567718b77034d294e46c36ede672456d159857d3e2"} Oct 30 13:33:48.381646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:48.381624 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" Oct 30 13:33:48.381748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:48.381640 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr" event={"ID":"9ad73dbe-40ec-4e07-a567-0d3b811063b6","Type":"ContainerDied","Data":"11bac4938abcfc8513a69864fc5ad9b2682ec44eafdd5460a03806fe0ea6a38c"} Oct 30 13:33:48.381748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:48.381685 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11bac4938abcfc8513a69864fc5ad9b2682ec44eafdd5460a03806fe0ea6a38c" Oct 30 13:33:48.383382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:48.383356 5049 generic.go:356] "Generic (PLEG): container finished" podID="99c279e4-7510-4613-865d-88b96986a27c" containerID="8d5e9a5e42162bb9c329ac428faf6e742fd8e051f0d2f9839be9f623afe5270a" exitCode=0 Oct 30 13:33:48.383481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:48.383404 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" event={"ID":"99c279e4-7510-4613-865d-88b96986a27c","Type":"ContainerDied","Data":"8d5e9a5e42162bb9c329ac428faf6e742fd8e051f0d2f9839be9f623afe5270a"} Oct 30 13:33:48.451515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:48.451451 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" podStartSLOduration=2.451433799 podStartE2EDuration="2.451433799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:48.448793441 +0000 UTC m=+2535.263987875" watchObservedRunningTime="2026-10-30 13:33:48.451433799 +0000 UTC m=+2535.266628221" Oct 30 13:33:49.092973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.092934 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:49.393689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.393595 5049 generic.go:356] "Generic (PLEG): container finished" podID="d981c28b-a864-4825-841f-df7978785cc8" containerID="f67687d9a123a0f33ef133567718b77034d294e46c36ede672456d159857d3e2" exitCode=0 Oct 30 13:33:49.394155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.393703 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" event={"ID":"d981c28b-a864-4825-841f-df7978785cc8","Type":"ContainerDied","Data":"f67687d9a123a0f33ef133567718b77034d294e46c36ede672456d159857d3e2"} Oct 30 13:33:49.593454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.593427 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:49.678430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.678344 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c279e4-7510-4613-865d-88b96986a27c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c279e4-7510-4613-865d-88b96986a27c-maas-api-openshift-ca\") pod \"99c279e4-7510-4613-865d-88b96986a27c\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " Oct 30 13:33:49.678430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.678422 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99c279e4-7510-4613-865d-88b96986a27c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c279e4-7510-4613-865d-88b96986a27c-maas-api-xks-ca\") pod \"99c279e4-7510-4613-865d-88b96986a27c\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " Oct 30 13:33:49.678616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.678539 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c279e4-7510-4613-865d-88b96986a27c-kube-api-access-w8w6l\" (UniqueName: \"kubernetes.io/projected/99c279e4-7510-4613-865d-88b96986a27c-kube-api-access-w8w6l\") pod \"99c279e4-7510-4613-865d-88b96986a27c\" (UID: \"99c279e4-7510-4613-865d-88b96986a27c\") " Oct 30 13:33:49.678836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.678791 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c279e4-7510-4613-865d-88b96986a27c-maas-api-openshift-ca" pod "99c279e4-7510-4613-865d-88b96986a27c" (UID: "99c279e4-7510-4613-865d-88b96986a27c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:49.679445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.679419 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c279e4-7510-4613-865d-88b96986a27c-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:49.681216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.681189 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c279e4-7510-4613-865d-88b96986a27c-maas-api-xks-ca" pod "99c279e4-7510-4613-865d-88b96986a27c" (UID: "99c279e4-7510-4613-865d-88b96986a27c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:33:49.681216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.681207 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c279e4-7510-4613-865d-88b96986a27c-kube-api-access-w8w6l" pod "99c279e4-7510-4613-865d-88b96986a27c" (UID: "99c279e4-7510-4613-865d-88b96986a27c"). InnerVolumeSpecName "kube-api-access-w8w6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:49.781228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.781192 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8w6l\" (UniqueName: \"kubernetes.io/projected/99c279e4-7510-4613-865d-88b96986a27c-kube-api-access-w8w6l\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:49.781228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:49.781225 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c279e4-7510-4613-865d-88b96986a27c-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:50.403668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.403639 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" Oct 30 13:33:50.404089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.403676 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj" event={"ID":"99c279e4-7510-4613-865d-88b96986a27c","Type":"ContainerDied","Data":"a3424188960fb88271c2fc60d8293df5974ea6d91dbe73d70b0ff8c4c5e09843"} Oct 30 13:33:50.404089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.403734 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3424188960fb88271c2fc60d8293df5974ea6d91dbe73d70b0ff8c4c5e09843" Oct 30 13:33:50.599485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.599459 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:50.695723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.695623 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d981c28b-a864-4825-841f-df7978785cc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d981c28b-a864-4825-841f-df7978785cc8-maas-api-xks-ca\") pod \"d981c28b-a864-4825-841f-df7978785cc8\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " Oct 30 13:33:50.695723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.695679 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d981c28b-a864-4825-841f-df7978785cc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d981c28b-a864-4825-841f-df7978785cc8-maas-api-openshift-ca\") pod \"d981c28b-a864-4825-841f-df7978785cc8\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " Oct 30 13:33:50.695966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.695782 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d981c28b-a864-4825-841f-df7978785cc8-kube-api-access-txdbd\" (UniqueName: \"kubernetes.io/projected/d981c28b-a864-4825-841f-df7978785cc8-kube-api-access-txdbd\") pod \"d981c28b-a864-4825-841f-df7978785cc8\" (UID: \"d981c28b-a864-4825-841f-df7978785cc8\") " Oct 30 13:33:50.696562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.696484 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d981c28b-a864-4825-841f-df7978785cc8-maas-api-openshift-ca" pod "d981c28b-a864-4825-841f-df7978785cc8" (UID: "d981c28b-a864-4825-841f-df7978785cc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:50.698934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.698907 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d981c28b-a864-4825-841f-df7978785cc8-kube-api-access-txdbd" pod "d981c28b-a864-4825-841f-df7978785cc8" (UID: "d981c28b-a864-4825-841f-df7978785cc8"). InnerVolumeSpecName "kube-api-access-txdbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:50.699085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.699061 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d981c28b-a864-4825-841f-df7978785cc8-maas-api-xks-ca" pod "d981c28b-a864-4825-841f-df7978785cc8" (UID: "d981c28b-a864-4825-841f-df7978785cc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:33:50.797498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.797471 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d981c28b-a864-4825-841f-df7978785cc8-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:50.797498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.797497 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d981c28b-a864-4825-841f-df7978785cc8-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:50.797498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:50.797507 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txdbd\" (UniqueName: \"kubernetes.io/projected/d981c28b-a864-4825-841f-df7978785cc8-kube-api-access-txdbd\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:51.002795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:51.002586 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:51.417904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:51.417870 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" event={"ID":"d981c28b-a864-4825-841f-df7978785cc8","Type":"ContainerDied","Data":"60b14c30a1a727339241e871a82cecedf284b82ff1b757a2d7fcd1c41b211397"} Oct 30 13:33:51.418284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:51.417924 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60b14c30a1a727339241e871a82cecedf284b82ff1b757a2d7fcd1c41b211397" Oct 30 13:33:51.418284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:51.417883 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz" Oct 30 13:33:52.828784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:52.827509 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:54.837398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.834464 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:54.843023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.842971 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw"] Oct 30 13:33:54.854081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.854041 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9ad73dbe-40ec-4e07-a567-0d3b811063b6" containerName="revoke-keys" Oct 30 13:33:54.854312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.854288 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad73dbe-40ec-4e07-a567-0d3b811063b6" containerName="revoke-keys" Oct 30 13:33:54.854496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.854466 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d981c28b-a864-4825-841f-df7978785cc8" containerName="revoke-keys" Oct 30 13:33:54.854496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.854476 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d981c28b-a864-4825-841f-df7978785cc8" containerName="revoke-keys" Oct 30 13:33:54.854496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.854494 5049 cpu_manager.go:414] "containerMap: removing container" podUID="99c279e4-7510-4613-865d-88b96986a27c" containerName="revoke-keys" Oct 30 13:33:54.854667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.854500 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c279e4-7510-4613-865d-88b96986a27c" containerName="revoke-keys" Oct 30 13:33:54.854936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.854916 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c279e4-7510-4613-865d-88b96986a27c" containerName="revoke-keys" Oct 30 13:33:54.855034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.854959 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad73dbe-40ec-4e07-a567-0d3b811063b6" containerName="revoke-keys" Oct 30 13:33:54.855034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.854987 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d981c28b-a864-4825-841f-df7978785cc8" containerName="revoke-keys" Oct 30 13:33:54.865406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.865379 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:54.872698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.870252 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw"] Oct 30 13:33:54.889162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.889131 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:33:54.889564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.889540 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:33:54.985279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.985240 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4sg4\" (UniqueName: \"kubernetes.io/projected/b558bfdf-310b-44ec-80f3-8a966f418f25-kube-api-access-p4sg4\") pod \"maas-api-revoke-sub-cbfd24a8-ft7vw\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:54.985478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.985290 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbfd24a8-ft7vw\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:54.985535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:54.985518 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbfd24a8-ft7vw\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:55.087149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:55.087111 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbfd24a8-ft7vw\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:55.087456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:55.087412 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4sg4\" (UniqueName: \"kubernetes.io/projected/b558bfdf-310b-44ec-80f3-8a966f418f25-kube-api-access-p4sg4\") pod \"maas-api-revoke-sub-cbfd24a8-ft7vw\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:55.087515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:55.087466 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbfd24a8-ft7vw\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:55.087878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:55.087859 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbfd24a8-ft7vw\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:55.089840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:55.089814 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbfd24a8-ft7vw\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:55.110072 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:55.110039 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4sg4\" (UniqueName: \"kubernetes.io/projected/b558bfdf-310b-44ec-80f3-8a966f418f25-kube-api-access-p4sg4\") pod \"maas-api-revoke-sub-cbfd24a8-ft7vw\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:55.189271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:55.189234 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:55.508939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:55.508907 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw"] Oct 30 13:33:55.510525 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:33:55.510496 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb558bfdf_310b_44ec_80f3_8a966f418f25.slice/crio-80e28bd84edccadb8d215e447ac8784c61c78e71e0ed9f3cee4e2ab775df26e4 WatchSource:0}: Error finding container 80e28bd84edccadb8d215e447ac8784c61c78e71e0ed9f3cee4e2ab775df26e4: Status 404 returned error can't find the container with id 80e28bd84edccadb8d215e447ac8784c61c78e71e0ed9f3cee4e2ab775df26e4 Oct 30 13:33:56.469995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.469964 5049 generic.go:356] "Generic (PLEG): container finished" podID="b558bfdf-310b-44ec-80f3-8a966f418f25" containerID="d876b2e4c2f540f8892d998eb00f17cc349a11182383dbbb5eb1bb9fc5993005" exitCode=0 Oct 30 13:33:56.470551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.470051 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" event={"ID":"b558bfdf-310b-44ec-80f3-8a966f418f25","Type":"ContainerDied","Data":"d876b2e4c2f540f8892d998eb00f17cc349a11182383dbbb5eb1bb9fc5993005"} Oct 30 13:33:56.470551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.470118 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" event={"ID":"b558bfdf-310b-44ec-80f3-8a966f418f25","Type":"ContainerStarted","Data":"80e28bd84edccadb8d215e447ac8784c61c78e71e0ed9f3cee4e2ab775df26e4"} Oct 30 13:33:56.589787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.589747 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4"] Oct 30 13:33:56.594257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.593098 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:56.599398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.599368 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.620124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.620089 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4"] Oct 30 13:33:56.694998 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:33:56.694914 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:33:56.712998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.712970 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338efa68-twqv4\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.713171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.713018 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338efa68-twqv4\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.713171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.713041 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lml2\" (UniqueName: \"kubernetes.io/projected/3a2207cc-2aa7-4552-83f6-ed428807529e-kube-api-access-2lml2\") pod \"maas-api-revoke-sub-338efa68-twqv4\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.814864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.814827 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338efa68-twqv4\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.814864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.814869 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338efa68-twqv4\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.815095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.814889 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lml2\" (UniqueName: \"kubernetes.io/projected/3a2207cc-2aa7-4552-83f6-ed428807529e-kube-api-access-2lml2\") pod \"maas-api-revoke-sub-338efa68-twqv4\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.815645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.815617 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338efa68-twqv4\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.817348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.817313 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338efa68-twqv4\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.838889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.838863 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lml2\" (UniqueName: \"kubernetes.io/projected/3a2207cc-2aa7-4552-83f6-ed428807529e-kube-api-access-2lml2\") pod \"maas-api-revoke-sub-338efa68-twqv4\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.923333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.923300 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:56.928314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.928290 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl"] Oct 30 13:33:56.933201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.933181 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:56.957405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:56.954424 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl"] Oct 30 13:33:57.017275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.017212 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bmbr\" (UniqueName: \"kubernetes.io/projected/08156865-c43b-4b9e-9231-be3922b57868-kube-api-access-2bmbr\") pod \"maas-api-revoke-sub-a75b2a72-gxdpl\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:57.017860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.017461 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08156865-c43b-4b9e-9231-be3922b57868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a75b2a72-gxdpl\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:57.017860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.017598 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08156865-c43b-4b9e-9231-be3922b57868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a75b2a72-gxdpl\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:57.118822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.118789 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08156865-c43b-4b9e-9231-be3922b57868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a75b2a72-gxdpl\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:57.119020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.118932 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08156865-c43b-4b9e-9231-be3922b57868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a75b2a72-gxdpl\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:57.119098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.119029 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bmbr\" (UniqueName: \"kubernetes.io/projected/08156865-c43b-4b9e-9231-be3922b57868-kube-api-access-2bmbr\") pod \"maas-api-revoke-sub-a75b2a72-gxdpl\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:57.119834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.119801 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08156865-c43b-4b9e-9231-be3922b57868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a75b2a72-gxdpl\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:57.121253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.121229 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08156865-c43b-4b9e-9231-be3922b57868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a75b2a72-gxdpl\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:57.140864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.140831 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bmbr\" (UniqueName: \"kubernetes.io/projected/08156865-c43b-4b9e-9231-be3922b57868-kube-api-access-2bmbr\") pod \"maas-api-revoke-sub-a75b2a72-gxdpl\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:57.257957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.257928 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4"] Oct 30 13:33:57.260717 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:33:57.260689 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2207cc_2aa7_4552_83f6_ed428807529e.slice/crio-0ef39ce6b37d2928e8195f894852dea043764bc20045b07f2cda8ae2bb457ca4 WatchSource:0}: Error finding container 0ef39ce6b37d2928e8195f894852dea043764bc20045b07f2cda8ae2bb457ca4: Status 404 returned error can't find the container with id 0ef39ce6b37d2928e8195f894852dea043764bc20045b07f2cda8ae2bb457ca4 Oct 30 13:33:57.283462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.283435 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:33:57.484496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.484448 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" event={"ID":"3a2207cc-2aa7-4552-83f6-ed428807529e","Type":"ContainerStarted","Data":"0ef39ce6b37d2928e8195f894852dea043764bc20045b07f2cda8ae2bb457ca4"} Oct 30 13:33:57.620498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.618439 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl"] Oct 30 13:33:57.643485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.643458 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:57.728424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.728384 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-xks-ca\") pod \"b558bfdf-310b-44ec-80f3-8a966f418f25\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " Oct 30 13:33:57.728695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.728673 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-openshift-ca\") pod \"b558bfdf-310b-44ec-80f3-8a966f418f25\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " Oct 30 13:33:57.728809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.728721 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b558bfdf-310b-44ec-80f3-8a966f418f25-kube-api-access-p4sg4\" (UniqueName: \"kubernetes.io/projected/b558bfdf-310b-44ec-80f3-8a966f418f25-kube-api-access-p4sg4\") pod \"b558bfdf-310b-44ec-80f3-8a966f418f25\" (UID: \"b558bfdf-310b-44ec-80f3-8a966f418f25\") " Oct 30 13:33:57.729151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.729115 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-openshift-ca" pod "b558bfdf-310b-44ec-80f3-8a966f418f25" (UID: "b558bfdf-310b-44ec-80f3-8a966f418f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:57.729441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.729420 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:57.731207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.731155 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-xks-ca" pod "b558bfdf-310b-44ec-80f3-8a966f418f25" (UID: "b558bfdf-310b-44ec-80f3-8a966f418f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:33:57.731569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.731541 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b558bfdf-310b-44ec-80f3-8a966f418f25-kube-api-access-p4sg4" pod "b558bfdf-310b-44ec-80f3-8a966f418f25" (UID: "b558bfdf-310b-44ec-80f3-8a966f418f25"). InnerVolumeSpecName "kube-api-access-p4sg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:57.830969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.830944 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4sg4\" (UniqueName: \"kubernetes.io/projected/b558bfdf-310b-44ec-80f3-8a966f418f25-kube-api-access-p4sg4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:57.830969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:57.830971 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b558bfdf-310b-44ec-80f3-8a966f418f25-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:58.417214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:58.417113 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:58.493796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:58.493748 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" Oct 30 13:33:58.493796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:58.493770 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw" event={"ID":"b558bfdf-310b-44ec-80f3-8a966f418f25","Type":"ContainerDied","Data":"80e28bd84edccadb8d215e447ac8784c61c78e71e0ed9f3cee4e2ab775df26e4"} Oct 30 13:33:58.494226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:58.493828 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80e28bd84edccadb8d215e447ac8784c61c78e71e0ed9f3cee4e2ab775df26e4" Oct 30 13:33:58.495689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:58.495663 5049 generic.go:356] "Generic (PLEG): container finished" podID="3a2207cc-2aa7-4552-83f6-ed428807529e" containerID="f67693126adce99073d49c1aa40351d83f3db7802185d16cdc0f14d14d9f4934" exitCode=0 Oct 30 13:33:58.495878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:58.495865 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" event={"ID":"3a2207cc-2aa7-4552-83f6-ed428807529e","Type":"ContainerDied","Data":"f67693126adce99073d49c1aa40351d83f3db7802185d16cdc0f14d14d9f4934"} Oct 30 13:33:58.497716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:58.497694 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" event={"ID":"08156865-c43b-4b9e-9231-be3922b57868","Type":"ContainerStarted","Data":"6e2bdb8d45f09af275e2039a0d109c8f8c2466937c7ea5b1b61fab53a5d66da9"} Oct 30 13:33:58.497802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:58.497731 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" event={"ID":"08156865-c43b-4b9e-9231-be3922b57868","Type":"ContainerStarted","Data":"857d711f161b39c097d4aef42bd74f93aa426a4e535d3fc99d7f17453a0c2ad0"} Oct 30 13:33:58.573550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:58.573496 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" podStartSLOduration=2.5734771199999997 podStartE2EDuration="2.57347712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:58.571195339 +0000 UTC m=+2545.386389774" watchObservedRunningTime="2026-10-30 13:33:58.57347712 +0000 UTC m=+2545.388671545" Oct 30 13:33:59.508076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.508038 5049 generic.go:356] "Generic (PLEG): container finished" podID="08156865-c43b-4b9e-9231-be3922b57868" containerID="6e2bdb8d45f09af275e2039a0d109c8f8c2466937c7ea5b1b61fab53a5d66da9" exitCode=0 Oct 30 13:33:59.508609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.508118 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" event={"ID":"08156865-c43b-4b9e-9231-be3922b57868","Type":"ContainerDied","Data":"6e2bdb8d45f09af275e2039a0d109c8f8c2466937c7ea5b1b61fab53a5d66da9"} Oct 30 13:33:59.709050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.709010 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:33:59.860353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.860308 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a2207cc-2aa7-4552-83f6-ed428807529e-kube-api-access-2lml2\" (UniqueName: \"kubernetes.io/projected/3a2207cc-2aa7-4552-83f6-ed428807529e-kube-api-access-2lml2\") pod \"3a2207cc-2aa7-4552-83f6-ed428807529e\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " Oct 30 13:33:59.860650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.860630 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-openshift-ca\") pod \"3a2207cc-2aa7-4552-83f6-ed428807529e\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " Oct 30 13:33:59.860736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.860684 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-xks-ca\") pod \"3a2207cc-2aa7-4552-83f6-ed428807529e\" (UID: \"3a2207cc-2aa7-4552-83f6-ed428807529e\") " Oct 30 13:33:59.861094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.861060 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-openshift-ca" pod "3a2207cc-2aa7-4552-83f6-ed428807529e" (UID: "3a2207cc-2aa7-4552-83f6-ed428807529e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:59.861522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.861497 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:59.863193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.862968 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-xks-ca" pod "3a2207cc-2aa7-4552-83f6-ed428807529e" (UID: "3a2207cc-2aa7-4552-83f6-ed428807529e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:33:59.863305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.863277 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2207cc-2aa7-4552-83f6-ed428807529e-kube-api-access-2lml2" pod "3a2207cc-2aa7-4552-83f6-ed428807529e" (UID: "3a2207cc-2aa7-4552-83f6-ed428807529e"). InnerVolumeSpecName "kube-api-access-2lml2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:59.962716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.962683 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lml2\" (UniqueName: \"kubernetes.io/projected/3a2207cc-2aa7-4552-83f6-ed428807529e-kube-api-access-2lml2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:33:59.962716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:33:59.962714 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2207cc-2aa7-4552-83f6-ed428807529e-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:00.223507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.222865 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:00.518397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.518275 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" Oct 30 13:34:00.518397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.518313 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4" event={"ID":"3a2207cc-2aa7-4552-83f6-ed428807529e","Type":"ContainerDied","Data":"0ef39ce6b37d2928e8195f894852dea043764bc20045b07f2cda8ae2bb457ca4"} Oct 30 13:34:00.518397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.518373 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ef39ce6b37d2928e8195f894852dea043764bc20045b07f2cda8ae2bb457ca4" Oct 30 13:34:00.713931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.713900 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:34:00.877272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.877236 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08156865-c43b-4b9e-9231-be3922b57868-kube-api-access-2bmbr\" (UniqueName: \"kubernetes.io/projected/08156865-c43b-4b9e-9231-be3922b57868-kube-api-access-2bmbr\") pod \"08156865-c43b-4b9e-9231-be3922b57868\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " Oct 30 13:34:00.877467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.877411 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08156865-c43b-4b9e-9231-be3922b57868-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08156865-c43b-4b9e-9231-be3922b57868-maas-api-xks-ca\") pod \"08156865-c43b-4b9e-9231-be3922b57868\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " Oct 30 13:34:00.877513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.877481 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08156865-c43b-4b9e-9231-be3922b57868-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08156865-c43b-4b9e-9231-be3922b57868-maas-api-openshift-ca\") pod \"08156865-c43b-4b9e-9231-be3922b57868\" (UID: \"08156865-c43b-4b9e-9231-be3922b57868\") " Oct 30 13:34:00.877942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.877904 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08156865-c43b-4b9e-9231-be3922b57868-maas-api-openshift-ca" pod "08156865-c43b-4b9e-9231-be3922b57868" (UID: "08156865-c43b-4b9e-9231-be3922b57868"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:00.878372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.878349 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08156865-c43b-4b9e-9231-be3922b57868-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:00.879563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.879537 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08156865-c43b-4b9e-9231-be3922b57868-maas-api-xks-ca" pod "08156865-c43b-4b9e-9231-be3922b57868" (UID: "08156865-c43b-4b9e-9231-be3922b57868"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:34:00.879699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.879559 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08156865-c43b-4b9e-9231-be3922b57868-kube-api-access-2bmbr" pod "08156865-c43b-4b9e-9231-be3922b57868" (UID: "08156865-c43b-4b9e-9231-be3922b57868"). InnerVolumeSpecName "kube-api-access-2bmbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:00.980406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.980370 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bmbr\" (UniqueName: \"kubernetes.io/projected/08156865-c43b-4b9e-9231-be3922b57868-kube-api-access-2bmbr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:00.980406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:00.980400 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08156865-c43b-4b9e-9231-be3922b57868-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:01.528125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:01.528089 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" event={"ID":"08156865-c43b-4b9e-9231-be3922b57868","Type":"ContainerDied","Data":"857d711f161b39c097d4aef42bd74f93aa426a4e535d3fc99d7f17453a0c2ad0"} Oct 30 13:34:01.528125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:01.528126 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl" Oct 30 13:34:01.528624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:01.528140 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="857d711f161b39c097d4aef42bd74f93aa426a4e535d3fc99d7f17453a0c2ad0" Oct 30 13:34:02.139516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:02.139476 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:04.252460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.252423 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:04.431958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.431920 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf"] Oct 30 13:34:04.432846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.432823 5049 cpu_manager.go:414] "containerMap: removing container" podUID="08156865-c43b-4b9e-9231-be3922b57868" containerName="revoke-keys" Oct 30 13:34:04.432846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.432845 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08156865-c43b-4b9e-9231-be3922b57868" containerName="revoke-keys" Oct 30 13:34:04.433566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.432871 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b558bfdf-310b-44ec-80f3-8a966f418f25" containerName="revoke-keys" Oct 30 13:34:04.433566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.432880 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b558bfdf-310b-44ec-80f3-8a966f418f25" containerName="revoke-keys" Oct 30 13:34:04.433566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.432942 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3a2207cc-2aa7-4552-83f6-ed428807529e" containerName="revoke-keys" Oct 30 13:34:04.433566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.432952 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a2207cc-2aa7-4552-83f6-ed428807529e" containerName="revoke-keys" Oct 30 13:34:04.433566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.433199 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a2207cc-2aa7-4552-83f6-ed428807529e" containerName="revoke-keys" Oct 30 13:34:04.433566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.433223 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="08156865-c43b-4b9e-9231-be3922b57868" containerName="revoke-keys" Oct 30 13:34:04.433566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.433238 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b558bfdf-310b-44ec-80f3-8a966f418f25" containerName="revoke-keys" Oct 30 13:34:04.437505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.437484 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.464938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.464901 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6da60a\"/\"kube-root-ca.crt\"" Oct 30 13:34:04.464938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.464901 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6da60a\"/\"openshift-service-ca.crt\"" Oct 30 13:34:04.465199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.465056 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6da60a\"/\"default-dockercfg-x29n8\"" Oct 30 13:34:04.465199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.465147 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6da60a\"/\"auth-test-model-6da60a-kserve-self-signed-certs\"" Oct 30 13:34:04.472057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.472032 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf"] Oct 30 13:34:04.536207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.536119 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-tmp-dir\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.536207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.536191 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3be1ca-961e-494a-8a13-68f3789695fd-tls-certs\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.536462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.536289 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-home\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.536530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.536480 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gljk6\" (UniqueName: \"kubernetes.io/projected/5f3be1ca-961e-494a-8a13-68f3789695fd-kube-api-access-gljk6\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.536591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.536533 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-dshm\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.536655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.536640 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-model-cache\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.557222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.557184 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:04.637600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.637565 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-tmp-dir\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.637794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.637640 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3be1ca-961e-494a-8a13-68f3789695fd-tls-certs\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.637794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.637730 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-home\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.637916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.637885 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gljk6\" (UniqueName: \"kubernetes.io/projected/5f3be1ca-961e-494a-8a13-68f3789695fd-kube-api-access-gljk6\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.637976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.637934 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-dshm\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.638034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.638017 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-model-cache\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.638079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.638034 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-tmp-dir\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.638120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.638096 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-home\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.638310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.638288 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-model-cache\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.640316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.640290 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-dshm\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.640838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.640813 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3be1ca-961e-494a-8a13-68f3789695fd-tls-certs\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.660178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.660150 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gljk6\" (UniqueName: \"kubernetes.io/projected/5f3be1ca-961e-494a-8a13-68f3789695fd-kube-api-access-gljk6\") pod \"auth-test-model-6da60a-kserve-79d4bd6885-qhmtf\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:04.775344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:04.771655 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:05.167690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:05.167657 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf"] Oct 30 13:34:05.170545 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:34:05.170519 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f3be1ca_961e_494a_8a13_68f3789695fd.slice/crio-eeb78c965106f4a18df32d941da93df956e48ad32184f4f10bc73bc59e0b5898 WatchSource:0}: Error finding container eeb78c965106f4a18df32d941da93df956e48ad32184f4f10bc73bc59e0b5898: Status 404 returned error can't find the container with id eeb78c965106f4a18df32d941da93df956e48ad32184f4f10bc73bc59e0b5898 Oct 30 13:34:05.568002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:05.567960 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" event={"ID":"5f3be1ca-961e-494a-8a13-68f3789695fd","Type":"ContainerStarted","Data":"438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f"} Oct 30 13:34:05.568493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:05.568010 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" event={"ID":"5f3be1ca-961e-494a-8a13-68f3789695fd","Type":"ContainerStarted","Data":"eeb78c965106f4a18df32d941da93df956e48ad32184f4f10bc73bc59e0b5898"} Oct 30 13:34:05.609209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:05.609158 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" podStartSLOduration=1.6091442630000001 podStartE2EDuration="1.609144263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:05.606916852 +0000 UTC m=+2552.422111276" watchObservedRunningTime="2026-10-30 13:34:05.609144263 +0000 UTC m=+2552.424338685" Oct 30 13:34:05.774780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:05.773477 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:06.280085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:06.280040 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:08.032348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:08.029133 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8"] Oct 30 13:34:08.116576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:08.116535 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:08.292515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:08.292412 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:10.105008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:10.104966 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:11.950279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:11.950245 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:13.967398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:13.967314 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:14.773847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:14.773816 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:14.786447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:14.786417 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:15.677257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:15.677226 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:34:15.994965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:15.994869 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:17.751811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:17.751775 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:17.911363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:17.911315 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:18.061989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:18.061959 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:19.579359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:19.573910 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:21.492045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:21.492005 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:23.063220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.063154 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" podUID="16efac8f-7d97-4b48-95d5-b1e8ed329020" containerName="main" containerID="cri-o://42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0" gracePeriod=45 Oct 30 13:34:23.289968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.289868 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:23.423144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.423117 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:34:23.480728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.480694 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16efac8f-7d97-4b48-95d5-b1e8ed329020-tls-certs\" (UniqueName: \"kubernetes.io/secret/16efac8f-7d97-4b48-95d5-b1e8ed329020-tls-certs\") pod \"16efac8f-7d97-4b48-95d5-b1e8ed329020\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " Oct 30 13:34:23.480728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.480740 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-home\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-home\") pod \"16efac8f-7d97-4b48-95d5-b1e8ed329020\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " Oct 30 13:34:23.481023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.480870 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-dshm\") pod \"16efac8f-7d97-4b48-95d5-b1e8ed329020\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " Oct 30 13:34:23.481023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.480934 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16efac8f-7d97-4b48-95d5-b1e8ed329020-kube-api-access-kkqcs\" (UniqueName: \"kubernetes.io/projected/16efac8f-7d97-4b48-95d5-b1e8ed329020-kube-api-access-kkqcs\") pod \"16efac8f-7d97-4b48-95d5-b1e8ed329020\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " Oct 30 13:34:23.481023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.480984 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-tmp-dir\") pod \"16efac8f-7d97-4b48-95d5-b1e8ed329020\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " Oct 30 13:34:23.481268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.481245 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-model-cache\") pod \"16efac8f-7d97-4b48-95d5-b1e8ed329020\" (UID: \"16efac8f-7d97-4b48-95d5-b1e8ed329020\") " Oct 30 13:34:23.481688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.481415 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-tmp-dir" pod "16efac8f-7d97-4b48-95d5-b1e8ed329020" (UID: "16efac8f-7d97-4b48-95d5-b1e8ed329020"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:23.481688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.481561 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-model-cache" pod "16efac8f-7d97-4b48-95d5-b1e8ed329020" (UID: "16efac8f-7d97-4b48-95d5-b1e8ed329020"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:23.482262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.482236 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:23.482385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.482269 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:23.482385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.482244 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-home" pod "16efac8f-7d97-4b48-95d5-b1e8ed329020" (UID: "16efac8f-7d97-4b48-95d5-b1e8ed329020"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:23.483565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.483520 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16efac8f-7d97-4b48-95d5-b1e8ed329020-tls-certs" pod "16efac8f-7d97-4b48-95d5-b1e8ed329020" (UID: "16efac8f-7d97-4b48-95d5-b1e8ed329020"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:34:23.483891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.483863 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16efac8f-7d97-4b48-95d5-b1e8ed329020-kube-api-access-kkqcs" pod "16efac8f-7d97-4b48-95d5-b1e8ed329020" (UID: "16efac8f-7d97-4b48-95d5-b1e8ed329020"). InnerVolumeSpecName "kube-api-access-kkqcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:23.484132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.484108 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-dshm" pod "16efac8f-7d97-4b48-95d5-b1e8ed329020" (UID: "16efac8f-7d97-4b48-95d5-b1e8ed329020"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:23.584226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.584190 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16efac8f-7d97-4b48-95d5-b1e8ed329020-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:23.584226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.584222 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:23.584226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.584230 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16efac8f-7d97-4b48-95d5-b1e8ed329020-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:23.584470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.584239 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkqcs\" (UniqueName: \"kubernetes.io/projected/16efac8f-7d97-4b48-95d5-b1e8ed329020-kube-api-access-kkqcs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:23.753061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.752962 5049 generic.go:356] "Generic (PLEG): container finished" podID="16efac8f-7d97-4b48-95d5-b1e8ed329020" containerID="42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0" exitCode=0 Oct 30 13:34:23.753061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.753002 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" event={"ID":"16efac8f-7d97-4b48-95d5-b1e8ed329020","Type":"ContainerDied","Data":"42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0"} Oct 30 13:34:23.753336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.753058 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" event={"ID":"16efac8f-7d97-4b48-95d5-b1e8ed329020","Type":"ContainerDied","Data":"36d875837e8f9f2b03ec72ef4459aca1932c2d605f5595eb1db4c3355c1d7ddd"} Oct 30 13:34:23.753336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.753085 5049 scope.go:122] "RemoveContainer" containerID="42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0" Oct 30 13:34:23.753336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.753015 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8" Oct 30 13:34:23.766313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.766295 5049 scope.go:122] "RemoveContainer" containerID="42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0" Oct 30 13:34:23.766617 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:34:23.766597 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0\": container with ID starting with 42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0 not found: ID does not exist" containerID="42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0" Oct 30 13:34:23.766661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.766625 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0"} err="failed to get container status \"42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0\": rpc error: code = NotFound desc = could not find container \"42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0\": container with ID starting with 42bb7638a5782a29d3bf84918c5d77220f3a26576405511ca043526e3ece6db0 not found: ID does not exist" Oct 30 13:34:23.803576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.803541 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8"] Oct 30 13:34:23.808916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:23.808881 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-9c6033/rate-test-model-9c6033-kserve-5b997cf465-tt5t8"] Oct 30 13:34:25.120675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:25.120643 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:25.338023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:25.337985 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16efac8f-7d97-4b48-95d5-b1e8ed329020" path="/var/lib/kubelet/pods/16efac8f-7d97-4b48-95d5-b1e8ed329020/volumes" Oct 30 13:34:26.680721 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:34:26.680685 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:34:27.008956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:27.007350 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:28.752422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:28.752387 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:29.126686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:29.126656 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:29.280501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:29.280465 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:30.495310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:30.495270 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:32.513855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:32.513818 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:34.618301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:34.618266 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:34.892071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:34.891977 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp"] Oct 30 13:34:34.966656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:34.966620 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:35.588461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:35.588428 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:36.447033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:36.446994 5049 scope.go:122] "RemoveContainer" containerID="b7375fce36f2c3a0017f760042dadf7b3bb27ed0c095456f330e8ca481d21fdd" Oct 30 13:34:36.459585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:36.459557 5049 scope.go:122] "RemoveContainer" containerID="00d25c7890a4316f8f5b513498e0a73a9781a50564be7361ff757391d6d11404" Oct 30 13:34:36.470165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:36.470140 5049 scope.go:122] "RemoveContainer" containerID="770386b00be09d5849427ee44f7bf29ae58266d42eeb424a9e5ca6b29a27c498" Oct 30 13:34:36.483107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:36.483086 5049 scope.go:122] "RemoveContainer" containerID="9ba827d5c6d365d01eb96b6d15af0f9216efbba5dafdbba612974b6e7d66b7e3" Oct 30 13:34:36.493544 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:36.493525 5049 scope.go:122] "RemoveContainer" containerID="584a04f21a23e1076c025641d45696c87f8f8d7a16840fa509ba1f968b64f546" Oct 30 13:34:36.820966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:36.820929 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:38.619121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:38.619084 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:38.773043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:38.773009 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:39.178939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:39.177534 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:39.551851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:39.551819 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:39.866233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:39.866145 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:39.870267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:39.870240 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj"] Oct 30 13:34:39.875341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:39.871556 5049 cpu_manager.go:414] "containerMap: removing container" podUID="16efac8f-7d97-4b48-95d5-b1e8ed329020" containerName="main" Oct 30 13:34:39.875341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:39.871583 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16efac8f-7d97-4b48-95d5-b1e8ed329020" containerName="main" Oct 30 13:34:39.875341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:39.871881 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="16efac8f-7d97-4b48-95d5-b1e8ed329020" containerName="main" Oct 30 13:34:39.880392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:39.878922 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:39.897283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:39.897260 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-428a0e-openshift-default-dockercfg-bqxbg\"" Oct 30 13:34:39.914934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:39.914907 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj"] Oct 30 13:34:40.047776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.047738 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk86p\" (UniqueName: \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-kube-api-access-dk86p\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.047963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.047797 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-podinfo\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.048013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.047965 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-envoy\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.048063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.048011 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-data\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.048115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.048098 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-certs\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.048206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.048191 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-socket\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.048306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.048293 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istiod-ca-cert\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.048428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.048416 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-token\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.048472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.048445 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-credential-socket\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.071895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.071855 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:40.154424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.150545 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-data\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.154424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.150627 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-certs\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.154424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.150710 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-socket\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.154424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.150838 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istiod-ca-cert\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.154424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.150982 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-token\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.154424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.151028 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-credential-socket\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.154424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.151089 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk86p\" (UniqueName: \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-kube-api-access-dk86p\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.154424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.151131 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-podinfo\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.156403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.155924 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-socket\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.174341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.167146 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-envoy\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.174341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.167736 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-data\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.174341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.167926 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-certs\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.191357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.186732 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-credential-socket\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.191357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.187389 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istiod-ca-cert\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.243339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.230672 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-podinfo\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.243339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.237394 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk86p\" (UniqueName: \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-kube-api-access-dk86p\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.257346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.255633 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-envoy\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.267191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.263879 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-token\") pod \"e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.303821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.303781 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:40.505550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.505453 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:40.853405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.853375 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj"] Oct 30 13:34:40.853551 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:34:40.853371 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc491c8a2_4e07_4851_aa5d_6a55dc247c59.slice/crio-25d40b0e1888c6cafb7d3271663c460206698c8b38a9759c0c52e2bc9ddb109c WatchSource:0}: Error finding container 25d40b0e1888c6cafb7d3271663c460206698c8b38a9759c0c52e2bc9ddb109c: Status 404 returned error can't find the container with id 25d40b0e1888c6cafb7d3271663c460206698c8b38a9759c0c52e2bc9ddb109c Oct 30 13:34:40.856389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.856360 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:40.856494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.856422 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:40.856494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:40.856455 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:41.008673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:41.008626 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" event={"ID":"c491c8a2-4e07-4851-aa5d-6a55dc247c59","Type":"ContainerStarted","Data":"2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5"} Oct 30 13:34:41.009169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:41.008702 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" event={"ID":"c491c8a2-4e07-4851-aa5d-6a55dc247c59","Type":"ContainerStarted","Data":"25d40b0e1888c6cafb7d3271663c460206698c8b38a9759c0c52e2bc9ddb109c"} Oct 30 13:34:41.105073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:41.104974 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:41.506461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:41.506369 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:41.512639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:41.512608 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:41.560439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:41.560388 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" podStartSLOduration=2.560373352 podStartE2EDuration="2.560373352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:41.067864538 +0000 UTC m=+2587.883058960" watchObservedRunningTime="2026-10-30 13:34:41.560373352 +0000 UTC m=+2588.375567774" Oct 30 13:34:41.943638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:41.943596 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c"] Oct 30 13:34:41.948758 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:41.948735 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:41.962684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:41.962661 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-082748-openshift-default-dockercfg-fhnpr\"" Oct 30 13:34:41.987951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:41.987915 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c"] Oct 30 13:34:42.022088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.022052 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.022601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.022138 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhrn\" (UniqueName: \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-kube-api-access-fbhrn\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.022601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.022172 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.022601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.022235 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.022601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.022280 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.022601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.022340 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.022601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.022380 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.022601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.022458 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.022601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.022509 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.023711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.023685 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:42.024819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.024799 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:34:42.129886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.129849 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhrn\" (UniqueName: \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-kube-api-access-fbhrn\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.130075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.129920 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.130075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.129989 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.130075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.130038 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.130236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.130086 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.130236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.130145 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.130236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.130226 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.130423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.130288 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.131662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.131631 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.132120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.132090 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.132790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.132705 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.133509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.133481 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.135542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.135518 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.136119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.136094 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.146121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.146091 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.147108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.147081 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.150958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.150933 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhrn\" (UniqueName: \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-kube-api-access-fbhrn\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.162219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.162196 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.265522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.265430 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:42.661235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.661198 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c"] Oct 30 13:34:42.663249 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:34:42.663211 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef45dd97_be25_4314_9a4b_08d5041c2dd5.slice/crio-d737c437158424b571c115dd81c398e09f1fde63103d24738d9eb16558fa3ac3 WatchSource:0}: Error finding container d737c437158424b571c115dd81c398e09f1fde63103d24738d9eb16558fa3ac3: Status 404 returned error can't find the container with id d737c437158424b571c115dd81c398e09f1fde63103d24738d9eb16558fa3ac3 Oct 30 13:34:42.666471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.666439 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:42.666553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.666522 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:42.666605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:42.666588 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:43.035401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.034749 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn"] Oct 30 13:34:43.040956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.040916 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" event={"ID":"ef45dd97-be25-4314-9a4b-08d5041c2dd5","Type":"ContainerStarted","Data":"a3382d1ff4f9c18b2af8e785505cc39217db7cfe1c3295eefccf791652f237aa"} Oct 30 13:34:43.041130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.041016 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2f4fda1-z86bn"] Oct 30 13:34:43.041130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.041044 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" event={"ID":"ef45dd97-be25-4314-9a4b-08d5041c2dd5","Type":"ContainerStarted","Data":"d737c437158424b571c115dd81c398e09f1fde63103d24738d9eb16558fa3ac3"} Oct 30 13:34:43.095856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.095794 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" podStartSLOduration=2.09577704 podStartE2EDuration="2.09577704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:43.095055705 +0000 UTC m=+2589.910250127" watchObservedRunningTime="2026-10-30 13:34:43.09577704 +0000 UTC m=+2589.910971464" Oct 30 13:34:43.266492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.266372 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:43.279348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.272780 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:43.285134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.281345 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 13:34:43.285134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.281427 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 13:34:43.346903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.346867 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf8ebcc-dc89-4279-97ca-e0d5d052cb18" path="/var/lib/kubelet/pods/daf8ebcc-dc89-4279-97ca-e0d5d052cb18/volumes" Oct 30 13:34:43.742409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.739909 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:43.929345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:43.922756 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:44.184912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:44.184878 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:44.266161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:44.266127 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 13:34:44.266401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:44.266178 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 13:34:44.483661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:44.483563 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:44.698279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:44.687224 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:45.266089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.266055 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 13:34:45.266663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.266116 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 13:34:45.442897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.442379 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6"] Oct 30 13:34:45.454048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.453979 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6"] Oct 30 13:34:45.454258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.454116 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.481116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.481092 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:34:45.481293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.481094 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:34:45.549780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.549747 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.550049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.549837 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx8cc\" (UniqueName: \"kubernetes.io/projected/c8f490aa-577a-47dd-847f-97739ad03531-kube-api-access-xx8cc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.550049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.549970 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.651400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.651342 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.651617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.651566 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.651692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.651630 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx8cc\" (UniqueName: \"kubernetes.io/projected/c8f490aa-577a-47dd-847f-97739ad03531-kube-api-access-xx8cc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.652853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.652827 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.655118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.655089 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.673123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.673097 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx8cc\" (UniqueName: \"kubernetes.io/projected/c8f490aa-577a-47dd-847f-97739ad03531-kube-api-access-xx8cc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.771195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.771167 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:45.862452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.856294 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb"] Oct 30 13:34:45.961296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:45.947918 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:46.162401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:46.160312 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:46.274385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:46.272528 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 13:34:46.274385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:46.272585 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 13:34:46.276707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:46.276675 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6"] Oct 30 13:34:46.277287 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:34:46.277263 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f490aa_577a_47dd_847f_97739ad03531.slice/crio-012a6702d44507b8103ebf71af5994d8f5de9791df033a4efb82b79849213da6 WatchSource:0}: Error finding container 012a6702d44507b8103ebf71af5994d8f5de9791df033a4efb82b79849213da6: Status 404 returned error can't find the container with id 012a6702d44507b8103ebf71af5994d8f5de9791df033a4efb82b79849213da6 Oct 30 13:34:47.057305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.057261 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:47.144927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.144873 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs"] Oct 30 13:34:47.150118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.150088 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:34:47.171868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.171839 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b4fb25-dockercfg-stfr8\"" Oct 30 13:34:47.172029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.171936 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b4fb25\"" Oct 30 13:34:47.174223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.174202 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs"] Oct 30 13:34:47.176699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.176673 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" event={"ID":"c8f490aa-577a-47dd-847f-97739ad03531","Type":"ContainerStarted","Data":"04497c44236e0af3f73c2cfa4469a5cf3accfe46f4fd3ea421e8c06a15bb7f21"} Oct 30 13:34:47.176820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.176718 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" event={"ID":"c8f490aa-577a-47dd-847f-97739ad03531","Type":"ContainerStarted","Data":"012a6702d44507b8103ebf71af5994d8f5de9791df033a4efb82b79849213da6"} Oct 30 13:34:47.256761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.256690 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:47.275373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.268793 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 13:34:47.275373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.268869 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 13:34:47.308419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.304084 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7tn\" (UniqueName: \"kubernetes.io/projected/0031597f-732c-4d8c-bada-e6c47ce81c42-kube-api-access-fc7tn\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs\" (UID: \"0031597f-732c-4d8c-bada-e6c47ce81c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:34:47.308419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.304455 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0031597f-732c-4d8c-bada-e6c47ce81c42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs\" (UID: \"0031597f-732c-4d8c-bada-e6c47ce81c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:34:47.585386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.582204 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7tn\" (UniqueName: \"kubernetes.io/projected/0031597f-732c-4d8c-bada-e6c47ce81c42-kube-api-access-fc7tn\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs\" (UID: \"0031597f-732c-4d8c-bada-e6c47ce81c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:34:47.585386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.582465 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0031597f-732c-4d8c-bada-e6c47ce81c42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs\" (UID: \"0031597f-732c-4d8c-bada-e6c47ce81c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:34:47.585386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.583599 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0031597f-732c-4d8c-bada-e6c47ce81c42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs\" (UID: \"0031597f-732c-4d8c-bada-e6c47ce81c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:34:47.605177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.605152 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7tn\" (UniqueName: \"kubernetes.io/projected/0031597f-732c-4d8c-bada-e6c47ce81c42-kube-api-access-fc7tn\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs\" (UID: \"0031597f-732c-4d8c-bada-e6c47ce81c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:34:47.773566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:47.773534 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:34:48.142494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.124118 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" podStartSLOduration=3.124098342 podStartE2EDuration="3.124098342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:47.255873862 +0000 UTC m=+2594.071068286" watchObservedRunningTime="2026-10-30 13:34:48.124098342 +0000 UTC m=+2594.939292767" Oct 30 13:34:48.146017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.145977 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs"] Oct 30 13:34:48.191180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.191147 5049 generic.go:356] "Generic (PLEG): container finished" podID="c8f490aa-577a-47dd-847f-97739ad03531" containerID="04497c44236e0af3f73c2cfa4469a5cf3accfe46f4fd3ea421e8c06a15bb7f21" exitCode=0 Oct 30 13:34:48.191567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.191535 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" event={"ID":"c8f490aa-577a-47dd-847f-97739ad03531","Type":"ContainerDied","Data":"04497c44236e0af3f73c2cfa4469a5cf3accfe46f4fd3ea421e8c06a15bb7f21"} Oct 30 13:34:48.193831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.193806 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" event={"ID":"0031597f-732c-4d8c-bada-e6c47ce81c42","Type":"ContainerStarted","Data":"af4c55f6335c24307d1379a7aa84cdb4b3c548b6a4b8715ea628a0bb0690522c"} Oct 30 13:34:48.266784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.266752 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 13:34:48.267097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.266812 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 13:34:48.445055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.444972 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:48.579996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.579958 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5"] Oct 30 13:34:48.586743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.586719 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:34:48.598123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.598089 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e43b89-8a7c-4a7b-9094-51364aa7bd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5\" (UID: \"73e43b89-8a7c-4a7b-9094-51364aa7bd85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:34:48.598376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.598301 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwcjk\" (UniqueName: \"kubernetes.io/projected/73e43b89-8a7c-4a7b-9094-51364aa7bd85-kube-api-access-wwcjk\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5\" (UID: \"73e43b89-8a7c-4a7b-9094-51364aa7bd85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:34:48.604785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.604756 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5"] Oct 30 13:34:48.701505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.659640 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-428a0e-dockercfg-r866k\"" Oct 30 13:34:48.701505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.659743 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-428a0e\"" Oct 30 13:34:48.701505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.688247 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:48.728378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.722563 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwcjk\" (UniqueName: \"kubernetes.io/projected/73e43b89-8a7c-4a7b-9094-51364aa7bd85-kube-api-access-wwcjk\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5\" (UID: \"73e43b89-8a7c-4a7b-9094-51364aa7bd85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:34:48.728378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.722814 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e43b89-8a7c-4a7b-9094-51364aa7bd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5\" (UID: \"73e43b89-8a7c-4a7b-9094-51364aa7bd85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:34:48.728378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.723819 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e43b89-8a7c-4a7b-9094-51364aa7bd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5\" (UID: \"73e43b89-8a7c-4a7b-9094-51364aa7bd85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:34:48.864339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:48.859125 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwcjk\" (UniqueName: \"kubernetes.io/projected/73e43b89-8a7c-4a7b-9094-51364aa7bd85-kube-api-access-wwcjk\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5\" (UID: \"73e43b89-8a7c-4a7b-9094-51364aa7bd85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:34:49.005369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.005142 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:34:49.204848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.204807 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" event={"ID":"0031597f-732c-4d8c-bada-e6c47ce81c42","Type":"ContainerStarted","Data":"46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757"} Oct 30 13:34:49.246562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.246371 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" podStartSLOduration=2.246349136 podStartE2EDuration="2.246349136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:49.241018991 +0000 UTC m=+2596.056213414" watchObservedRunningTime="2026-10-30 13:34:49.246349136 +0000 UTC m=+2596.061543563" Oct 30 13:34:49.267221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.267141 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 13:34:49.267221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.267194 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 13:34:49.444686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.444227 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5"] Oct 30 13:34:49.446424 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:34:49.445842 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e43b89_8a7c_4a7b_9094_51364aa7bd85.slice/crio-2e9644a12c6c7b24c1d4fb0090508745f2aa59902ec65864eaaf42d76929aa67 WatchSource:0}: Error finding container 2e9644a12c6c7b24c1d4fb0090508745f2aa59902ec65864eaaf42d76929aa67: Status 404 returned error can't find the container with id 2e9644a12c6c7b24c1d4fb0090508745f2aa59902ec65864eaaf42d76929aa67 Oct 30 13:34:49.467462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.467440 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:49.594022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.593989 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-xks-ca\") pod \"c8f490aa-577a-47dd-847f-97739ad03531\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " Oct 30 13:34:49.594217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.594111 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f490aa-577a-47dd-847f-97739ad03531-kube-api-access-xx8cc\" (UniqueName: \"kubernetes.io/projected/c8f490aa-577a-47dd-847f-97739ad03531-kube-api-access-xx8cc\") pod \"c8f490aa-577a-47dd-847f-97739ad03531\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " Oct 30 13:34:49.594217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.594193 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-openshift-ca\") pod \"c8f490aa-577a-47dd-847f-97739ad03531\" (UID: \"c8f490aa-577a-47dd-847f-97739ad03531\") " Oct 30 13:34:49.594753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.594594 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-openshift-ca" pod "c8f490aa-577a-47dd-847f-97739ad03531" (UID: "c8f490aa-577a-47dd-847f-97739ad03531"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:49.594897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.594878 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:49.596203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.596182 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f490aa-577a-47dd-847f-97739ad03531-kube-api-access-xx8cc" pod "c8f490aa-577a-47dd-847f-97739ad03531" (UID: "c8f490aa-577a-47dd-847f-97739ad03531"). InnerVolumeSpecName "kube-api-access-xx8cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:49.596515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.596502 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-xks-ca" pod "c8f490aa-577a-47dd-847f-97739ad03531" (UID: "c8f490aa-577a-47dd-847f-97739ad03531"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:34:49.696288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.696256 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx8cc\" (UniqueName: \"kubernetes.io/projected/c8f490aa-577a-47dd-847f-97739ad03531-kube-api-access-xx8cc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:49.696288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.696285 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f490aa-577a-47dd-847f-97739ad03531-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:49.918375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:49.918331 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" podUID="936e185a-1af8-4104-a8fb-ab6c2c291e60" containerName="main" containerID="cri-o://429d936f990e6bfde7480e58927b7ae5fd690af40c85382a796276bd9deb6460" gracePeriod=45 Oct 30 13:34:50.219400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.219359 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" event={"ID":"73e43b89-8a7c-4a7b-9094-51364aa7bd85","Type":"ContainerStarted","Data":"eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30"} Oct 30 13:34:50.219548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.219429 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" event={"ID":"73e43b89-8a7c-4a7b-9094-51364aa7bd85","Type":"ContainerStarted","Data":"2e9644a12c6c7b24c1d4fb0090508745f2aa59902ec65864eaaf42d76929aa67"} Oct 30 13:34:50.219777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.219755 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:34:50.221720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.221693 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" event={"ID":"c8f490aa-577a-47dd-847f-97739ad03531","Type":"ContainerDied","Data":"012a6702d44507b8103ebf71af5994d8f5de9791df033a4efb82b79849213da6"} Oct 30 13:34:50.221869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.221747 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012a6702d44507b8103ebf71af5994d8f5de9791df033a4efb82b79849213da6" Oct 30 13:34:50.221869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.221697 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6" Oct 30 13:34:50.254959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.254909 5049 generic.go:356] "Generic (PLEG): container finished" podID="936e185a-1af8-4104-a8fb-ab6c2c291e60" containerID="429d936f990e6bfde7480e58927b7ae5fd690af40c85382a796276bd9deb6460" exitCode=0 Oct 30 13:34:50.255548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.255125 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" event={"ID":"936e185a-1af8-4104-a8fb-ab6c2c291e60","Type":"ContainerDied","Data":"429d936f990e6bfde7480e58927b7ae5fd690af40c85382a796276bd9deb6460"} Oct 30 13:34:50.255548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.255175 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" event={"ID":"936e185a-1af8-4104-a8fb-ab6c2c291e60","Type":"ContainerDied","Data":"a070eedac25c167f27115c75b9dfd61d3256fcfc8069bd112566d16d4b579172"} Oct 30 13:34:50.255548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.255192 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a070eedac25c167f27115c75b9dfd61d3256fcfc8069bd112566d16d4b579172" Oct 30 13:34:50.255548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.255350 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:34:50.260611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.260588 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:34:50.271297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.271272 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:50.271577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.271559 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:50.272314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.272294 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:34:50.306276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.306242 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-home\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-home\") pod \"936e185a-1af8-4104-a8fb-ab6c2c291e60\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " Oct 30 13:34:50.306417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.306348 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-tmp-dir\") pod \"936e185a-1af8-4104-a8fb-ab6c2c291e60\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " Oct 30 13:34:50.306520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.306470 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-dshm\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-dshm\") pod \"936e185a-1af8-4104-a8fb-ab6c2c291e60\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " Oct 30 13:34:50.306520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.306507 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-model-cache\") pod \"936e185a-1af8-4104-a8fb-ab6c2c291e60\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " Oct 30 13:34:50.306648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.306564 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-home" pod "936e185a-1af8-4104-a8fb-ab6c2c291e60" (UID: "936e185a-1af8-4104-a8fb-ab6c2c291e60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:50.306818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.306764 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-model-cache" pod "936e185a-1af8-4104-a8fb-ab6c2c291e60" (UID: "936e185a-1af8-4104-a8fb-ab6c2c291e60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:50.306818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.306766 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936e185a-1af8-4104-a8fb-ab6c2c291e60-kube-api-access-b2mdz\" (UniqueName: \"kubernetes.io/projected/936e185a-1af8-4104-a8fb-ab6c2c291e60-kube-api-access-b2mdz\") pod \"936e185a-1af8-4104-a8fb-ab6c2c291e60\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " Oct 30 13:34:50.306978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.306818 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936e185a-1af8-4104-a8fb-ab6c2c291e60-tls-certs\" (UniqueName: \"kubernetes.io/secret/936e185a-1af8-4104-a8fb-ab6c2c291e60-tls-certs\") pod \"936e185a-1af8-4104-a8fb-ab6c2c291e60\" (UID: \"936e185a-1af8-4104-a8fb-ab6c2c291e60\") " Oct 30 13:34:50.307430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.307406 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-tmp-dir" pod "936e185a-1af8-4104-a8fb-ab6c2c291e60" (UID: "936e185a-1af8-4104-a8fb-ab6c2c291e60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:50.307778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.307753 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:50.307848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.307782 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:50.307848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.307796 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:50.310939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.310906 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-dshm" pod "936e185a-1af8-4104-a8fb-ab6c2c291e60" (UID: "936e185a-1af8-4104-a8fb-ab6c2c291e60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:50.311474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.311446 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936e185a-1af8-4104-a8fb-ab6c2c291e60-kube-api-access-b2mdz" pod "936e185a-1af8-4104-a8fb-ab6c2c291e60" (UID: "936e185a-1af8-4104-a8fb-ab6c2c291e60"). InnerVolumeSpecName "kube-api-access-b2mdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:50.312001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.311978 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936e185a-1af8-4104-a8fb-ab6c2c291e60-tls-certs" pod "936e185a-1af8-4104-a8fb-ab6c2c291e60" (UID: "936e185a-1af8-4104-a8fb-ab6c2c291e60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:34:50.319368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.319307 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" podStartSLOduration=2.319290569 podStartE2EDuration="2.319290569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:50.307412787 +0000 UTC m=+2597.122607211" watchObservedRunningTime="2026-10-30 13:34:50.319290569 +0000 UTC m=+2597.134484993" Oct 30 13:34:50.411974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.411934 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2mdz\" (UniqueName: \"kubernetes.io/projected/936e185a-1af8-4104-a8fb-ab6c2c291e60-kube-api-access-b2mdz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:50.411974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.411971 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936e185a-1af8-4104-a8fb-ab6c2c291e60-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:50.412241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.411986 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936e185a-1af8-4104-a8fb-ab6c2c291e60-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:34:50.745894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:50.745862 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:51.263884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:51.263849 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp" Oct 30 13:34:51.317272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:51.317233 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp"] Oct 30 13:34:51.323927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:51.323882 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f0a31b/auth-test-model-f0a31b-kserve-d78d4774f-bvkfp"] Oct 30 13:34:51.339890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:51.339860 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936e185a-1af8-4104-a8fb-ab6c2c291e60" path="/var/lib/kubelet/pods/936e185a-1af8-4104-a8fb-ab6c2c291e60/volumes" Oct 30 13:34:52.807503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:52.807464 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:53.040553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:53.040513 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5"] Oct 30 13:34:53.051358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:53.051312 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cbda8b-wv8z5"] Oct 30 13:34:53.340779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:53.340744 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df37af6c-fd30-44c3-bc82-2b4166f0825d" path="/var/lib/kubelet/pods/df37af6c-fd30-44c3-bc82-2b4166f0825d/volumes" Oct 30 13:34:54.501210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:54.501173 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:55.790420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:55.790387 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7"] Oct 30 13:34:55.813115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:55.813083 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz"] Oct 30 13:34:55.960257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:55.960226 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:56.134287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:56.134240 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:56.300594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:56.300561 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:56.682874 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:34:56.682841 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:34:58.269815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:58.269775 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:59.677381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:34:59.677341 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:00.014542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:00.014450 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:00.791679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:00.791639 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" podUID="5c5aa42c-f531-44af-9bd7-9e2d600d6a1c" containerName="payload-processing" containerID="cri-o://31c7b02ea9a9e9a1aecf62bd116e11820c94c339c0aacab39a460ccb2023857d" gracePeriod=25 Oct 30 13:35:00.814170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:00.814123 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" podUID="e1fb92c3-28ac-4223-acee-0cc0b8100506" containerName="payload-pre-processing" containerID="cri-o://fbb41365b1db700892acdf302872986c99980bef4a0ec80a6fcaae2ee5751948" gracePeriod=25 Oct 30 13:35:00.894178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:00.894130 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" podUID="9e2b7bd0-8626-4025-bc7e-546b26a77764" containerName="main" containerID="cri-o://0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a" gracePeriod=45 Oct 30 13:35:01.026805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.025043 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:01.266666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.266634 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:35:01.266866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.266717 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:35:01.266866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.266762 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:35:01.352287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.352199 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-home\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-home\") pod \"9e2b7bd0-8626-4025-bc7e-546b26a77764\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " Oct 30 13:35:01.352478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.352440 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2b7bd0-8626-4025-bc7e-546b26a77764-kube-api-access-56dk4\" (UniqueName: \"kubernetes.io/projected/9e2b7bd0-8626-4025-bc7e-546b26a77764-kube-api-access-56dk4\") pod \"9e2b7bd0-8626-4025-bc7e-546b26a77764\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " Oct 30 13:35:01.352551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.352521 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-model-cache\") pod \"9e2b7bd0-8626-4025-bc7e-546b26a77764\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " Oct 30 13:35:01.352606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.352554 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-tmp-dir\") pod \"9e2b7bd0-8626-4025-bc7e-546b26a77764\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " Oct 30 13:35:01.352606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.352548 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-home" pod "9e2b7bd0-8626-4025-bc7e-546b26a77764" (UID: "9e2b7bd0-8626-4025-bc7e-546b26a77764"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:01.352782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.352709 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e2b7bd0-8626-4025-bc7e-546b26a77764-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2b7bd0-8626-4025-bc7e-546b26a77764-tls-certs\") pod \"9e2b7bd0-8626-4025-bc7e-546b26a77764\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " Oct 30 13:35:01.352782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.352754 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-dshm\") pod \"9e2b7bd0-8626-4025-bc7e-546b26a77764\" (UID: \"9e2b7bd0-8626-4025-bc7e-546b26a77764\") " Oct 30 13:35:01.353116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.353092 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-tmp-dir" pod "9e2b7bd0-8626-4025-bc7e-546b26a77764" (UID: "9e2b7bd0-8626-4025-bc7e-546b26a77764"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:01.353584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.353407 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-model-cache" pod "9e2b7bd0-8626-4025-bc7e-546b26a77764" (UID: "9e2b7bd0-8626-4025-bc7e-546b26a77764"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:01.353584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.353566 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:01.353759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.353589 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:01.353759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.353602 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:01.355574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.355553 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-dshm" pod "9e2b7bd0-8626-4025-bc7e-546b26a77764" (UID: "9e2b7bd0-8626-4025-bc7e-546b26a77764"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:01.355783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.355754 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2b7bd0-8626-4025-bc7e-546b26a77764-tls-certs" pod "9e2b7bd0-8626-4025-bc7e-546b26a77764" (UID: "9e2b7bd0-8626-4025-bc7e-546b26a77764"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:01.356059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.356039 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2b7bd0-8626-4025-bc7e-546b26a77764-kube-api-access-56dk4" pod "9e2b7bd0-8626-4025-bc7e-546b26a77764" (UID: "9e2b7bd0-8626-4025-bc7e-546b26a77764"). InnerVolumeSpecName "kube-api-access-56dk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:01.373843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.373809 5049 generic.go:356] "Generic (PLEG): container finished" podID="9e2b7bd0-8626-4025-bc7e-546b26a77764" containerID="0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a" exitCode=0 Oct 30 13:35:01.373979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.373869 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" Oct 30 13:35:01.374042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.374005 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" event={"ID":"9e2b7bd0-8626-4025-bc7e-546b26a77764","Type":"ContainerDied","Data":"0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a"} Oct 30 13:35:01.374042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.374039 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb" event={"ID":"9e2b7bd0-8626-4025-bc7e-546b26a77764","Type":"ContainerDied","Data":"7fdc97c9f1e221e7ee6f27164750082ae6c29edc2442708c0616211c403e8176"} Oct 30 13:35:01.374169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.374056 5049 scope.go:122] "RemoveContainer" containerID="0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a" Oct 30 13:35:01.413508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.411534 5049 scope.go:122] "RemoveContainer" containerID="0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a" Oct 30 13:35:01.413508 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:35:01.411922 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a\": container with ID starting with 0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a not found: ID does not exist" containerID="0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a" Oct 30 13:35:01.413508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.411964 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a"} err="failed to get container status \"0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a\": rpc error: code = NotFound desc = could not find container \"0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a\": container with ID starting with 0dad0062203a1aa11834357e0cdf39c65fded199deccabbee36ea444c2e2b64a not found: ID does not exist" Oct 30 13:35:01.455475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.455442 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2b7bd0-8626-4025-bc7e-546b26a77764-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:01.455475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.455474 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2b7bd0-8626-4025-bc7e-546b26a77764-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:01.455744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.455490 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56dk4\" (UniqueName: \"kubernetes.io/projected/9e2b7bd0-8626-4025-bc7e-546b26a77764-kube-api-access-56dk4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:01.474519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.472207 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb"] Oct 30 13:35:01.478798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:01.478769 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-fdd4d4/auth-test-model-fdd4d4-kserve-77cd5b88c7-5fhjb"] Oct 30 13:35:03.199210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:03.199171 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:03.340235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:03.340198 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2b7bd0-8626-4025-bc7e-546b26a77764" path="/var/lib/kubelet/pods/9e2b7bd0-8626-4025-bc7e-546b26a77764/volumes" Oct 30 13:35:04.982782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:04.982748 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:06.804868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:06.804824 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:08.149694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:08.149658 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" podUID="5c5aa42c-f531-44af-9bd7-9e2d600d6a1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:35:08.150215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:08.149653 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" podUID="e1fb92c3-28ac-4223-acee-0cc0b8100506" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:35:08.712533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:08.709970 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:10.414401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:10.407232 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:12.334813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:12.334048 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:14.135562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:14.135517 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:15.733419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:15.733375 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:15.927194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:15.927161 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:16.164340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.163970 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf"] Oct 30 13:35:16.205788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.205751 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx"] Oct 30 13:35:16.212851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.212806 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c8f490aa-577a-47dd-847f-97739ad03531" containerName="revoke-keys" Oct 30 13:35:16.213081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.213061 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f490aa-577a-47dd-847f-97739ad03531" containerName="revoke-keys" Oct 30 13:35:16.213285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.213271 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9e2b7bd0-8626-4025-bc7e-546b26a77764" containerName="main" Oct 30 13:35:16.213604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.213588 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2b7bd0-8626-4025-bc7e-546b26a77764" containerName="main" Oct 30 13:35:16.213773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.213759 5049 cpu_manager.go:414] "containerMap: removing container" podUID="936e185a-1af8-4104-a8fb-ab6c2c291e60" containerName="main" Oct 30 13:35:16.213893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.213881 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936e185a-1af8-4104-a8fb-ab6c2c291e60" containerName="main" Oct 30 13:35:16.214307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.214276 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2b7bd0-8626-4025-bc7e-546b26a77764" containerName="main" Oct 30 13:35:16.214510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.214477 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="936e185a-1af8-4104-a8fb-ab6c2c291e60" containerName="main" Oct 30 13:35:16.214673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.214662 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f490aa-577a-47dd-847f-97739ad03531" containerName="revoke-keys" Oct 30 13:35:16.250683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.249155 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.265360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.264958 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx"] Oct 30 13:35:16.269150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.267549 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2b254d-openshift-default-dockercfg-wfzql\"" Oct 30 13:35:16.272510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.272203 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:16.294404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.294371 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-certs\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.294592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.294465 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-data\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.294592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.294506 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-socket\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.294592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.294544 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-token\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.294743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.294726 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-credential-socket\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.294850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.294820 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-envoy\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.294894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.294878 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istiod-ca-cert\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.294942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.294901 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-podinfo\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.294942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.294927 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv94f\" (UniqueName: \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-kube-api-access-zv94f\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.454463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.422965 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-credential-socket\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.454463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.423119 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-envoy\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.454463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.423573 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istiod-ca-cert\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.454463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.423652 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-podinfo\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.454463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.423722 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv94f\" (UniqueName: \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-kube-api-access-zv94f\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.454463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.453270 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-certs\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.479749 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.455918 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-credential-socket\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.479749 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.460916 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-data\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.479749 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.479634 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-certs\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.490450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.453650 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-data\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.522866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.491590 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istiod-ca-cert\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.522866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.518681 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-socket\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.522866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.518745 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-token\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.522866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.519905 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-socket\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.534768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.533864 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-envoy\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.593690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.592389 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-token\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.593690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.592522 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv94f\" (UniqueName: \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-kube-api-access-zv94f\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.594979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.594950 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:16.597962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.597939 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-podinfo\") pod \"e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.825233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.825190 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:16.882251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.882203 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:16.989729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:16.989699 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:17.282959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.282933 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx"] Oct 30 13:35:17.283733 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:17.283707 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6b69e3_e48f_4eaa_960d_e149e93cd038.slice/crio-07f8cade572edbcb19f6ac2b8169d10dabb3146a1c918fe8236ba2061e84be80 WatchSource:0}: Error finding container 07f8cade572edbcb19f6ac2b8169d10dabb3146a1c918fe8236ba2061e84be80: Status 404 returned error can't find the container with id 07f8cade572edbcb19f6ac2b8169d10dabb3146a1c918fe8236ba2061e84be80 Oct 30 13:35:17.286789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.286756 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:17.286884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.286831 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:17.286884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.286861 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:17.468411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.468368 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv"] Oct 30 13:35:17.492207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.492176 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.529596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.527450 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-cf3c79-openshift-default-dockercfg-kqb46\"" Oct 30 13:35:17.534354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.529980 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv"] Oct 30 13:35:17.541400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.541358 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r22b9\" (UniqueName: \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-kube-api-access-r22b9\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.541551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.541423 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-envoy\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.541551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.541493 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-certs\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.541551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.541538 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istiod-ca-cert\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.541729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.541700 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-credential-socket\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.541812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.541797 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-token\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.541876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.541862 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-podinfo\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.541937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.541918 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-data\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.541994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.541968 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-socket\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.842421 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-credential-socket\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.842552 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-token\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.842623 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-podinfo\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.842685 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-data\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.842741 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-socket\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.842812 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r22b9\" (UniqueName: \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-kube-api-access-r22b9\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.842856 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-envoy\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.842941 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-certs\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.842993 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istiod-ca-cert\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.843899 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istiod-ca-cert\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.844408 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-credential-socket\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.845016 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-certs\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.846088 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-data\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.852386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.846403 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-socket\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.873892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.873642 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:17.873892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.873841 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-token\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.877003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.876968 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-podinfo\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.878009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.877946 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r22b9\" (UniqueName: \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-kube-api-access-r22b9\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.898364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.898137 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-envoy\") pod \"e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:17.900538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.900499 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" event={"ID":"9a6b69e3-e48f-4eaa-960d-e149e93cd038","Type":"ContainerStarted","Data":"6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a"} Oct 30 13:35:17.900688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.900553 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" event={"ID":"9a6b69e3-e48f-4eaa-960d-e149e93cd038","Type":"ContainerStarted","Data":"07f8cade572edbcb19f6ac2b8169d10dabb3146a1c918fe8236ba2061e84be80"} Oct 30 13:35:17.943917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:17.943855 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" podStartSLOduration=1.943837686 podStartE2EDuration="1.943837686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:17.942711334 +0000 UTC m=+2624.757905792" watchObservedRunningTime="2026-10-30 13:35:17.943837686 +0000 UTC m=+2624.759032113" Oct 30 13:35:18.028949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.028913 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:18.133472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.133377 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:18.149608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.149576 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" podUID="e1fb92c3-28ac-4223-acee-0cc0b8100506" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:35:18.150400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.150367 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" podUID="5c5aa42c-f531-44af-9bd7-9e2d600d6a1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:35:18.299209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.298182 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:18.479509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.479482 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv"] Oct 30 13:35:18.481293 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:18.481268 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcdf8ca_4321_4ccf_ab10_a0e53be8fd36.slice/crio-1b08afe33f35722b8f80c7c92195b7935433da89b1340f408483478a256b7519 WatchSource:0}: Error finding container 1b08afe33f35722b8f80c7c92195b7935433da89b1340f408483478a256b7519: Status 404 returned error can't find the container with id 1b08afe33f35722b8f80c7c92195b7935433da89b1340f408483478a256b7519 Oct 30 13:35:18.484442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.484413 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:18.484528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.484476 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:18.484528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.484506 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:18.883033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.882995 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:18.889912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.889888 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:18.914930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.914895 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" event={"ID":"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36","Type":"ContainerStarted","Data":"f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e"} Oct 30 13:35:18.915120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.914959 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" event={"ID":"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36","Type":"ContainerStarted","Data":"1b08afe33f35722b8f80c7c92195b7935433da89b1340f408483478a256b7519"} Oct 30 13:35:18.915376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.915355 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:18.916448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.916419 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:35:18.997476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:18.997424 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" podStartSLOduration=1.997410898 podStartE2EDuration="1.997410898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:18.995547005 +0000 UTC m=+2625.810741428" watchObservedRunningTime="2026-10-30 13:35:18.997410898 +0000 UTC m=+2625.812605319" Oct 30 13:35:19.134046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:19.133961 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:19.140625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:19.140597 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:19.925235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:19.925204 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:19.926387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:19.926356 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:35:20.357414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:20.356188 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:20.585143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:20.585106 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:20.804781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:20.804742 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:21.334695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:21.318114 5049 generic.go:356] "Generic (PLEG): container finished" podID="e1fb92c3-28ac-4223-acee-0cc0b8100506" containerID="fbb41365b1db700892acdf302872986c99980bef4a0ec80a6fcaae2ee5751948" exitCode=0 Oct 30 13:35:21.334695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:21.318337 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" event={"ID":"e1fb92c3-28ac-4223-acee-0cc0b8100506","Type":"ContainerDied","Data":"fbb41365b1db700892acdf302872986c99980bef4a0ec80a6fcaae2ee5751948"} Oct 30 13:35:21.334695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:21.321195 5049 generic.go:356] "Generic (PLEG): container finished" podID="5c5aa42c-f531-44af-9bd7-9e2d600d6a1c" containerID="31c7b02ea9a9e9a1aecf62bd116e11820c94c339c0aacab39a460ccb2023857d" exitCode=0 Oct 30 13:35:21.334695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:21.322499 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" event={"ID":"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c","Type":"ContainerDied","Data":"31c7b02ea9a9e9a1aecf62bd116e11820c94c339c0aacab39a460ccb2023857d"} Oct 30 13:35:22.185655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.185613 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:35:22.272600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.272473 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-plugins-config-volume\") pod \"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c\" (UID: \"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c\") " Oct 30 13:35:22.273234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.273189 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-kube-api-access-kh9xf\" (UniqueName: \"kubernetes.io/projected/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-kube-api-access-kh9xf\") pod \"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c\" (UID: \"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c\") " Oct 30 13:35:22.386370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.300125 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-plugins-config-volume" pod "5c5aa42c-f531-44af-9bd7-9e2d600d6a1c" (UID: "5c5aa42c-f531-44af-9bd7-9e2d600d6a1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:22.386370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.337452 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" podUID="68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.20:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:35:22.386370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.378824 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:22.416382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.415779 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" event={"ID":"e1fb92c3-28ac-4223-acee-0cc0b8100506","Type":"ContainerDied","Data":"861715ce6c426936107ab027571029773f3ccaea1c2e331a2effafedac3377c0"} Oct 30 13:35:22.416382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.415841 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="861715ce6c426936107ab027571029773f3ccaea1c2e331a2effafedac3377c0" Oct 30 13:35:22.432391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.423979 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" event={"ID":"5c5aa42c-f531-44af-9bd7-9e2d600d6a1c","Type":"ContainerDied","Data":"ee05bb07d4026d342ad71f5ec3a8b366acca37c4c4d4ab66b3b680ce4821a1ff"} Oct 30 13:35:22.432391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.424057 5049 scope.go:122] "RemoveContainer" containerID="31c7b02ea9a9e9a1aecf62bd116e11820c94c339c0aacab39a460ccb2023857d" Oct 30 13:35:22.432391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.424066 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7" Oct 30 13:35:22.432391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.427853 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:35:22.452631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.451519 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-kube-api-access-kh9xf" pod "5c5aa42c-f531-44af-9bd7-9e2d600d6a1c" (UID: "5c5aa42c-f531-44af-9bd7-9e2d600d6a1c"). InnerVolumeSpecName "kube-api-access-kh9xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:22.521496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.519250 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:22.523976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.521932 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1fb92c3-28ac-4223-acee-0cc0b8100506-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fb92c3-28ac-4223-acee-0cc0b8100506-plugins-config-volume\") pod \"e1fb92c3-28ac-4223-acee-0cc0b8100506\" (UID: \"e1fb92c3-28ac-4223-acee-0cc0b8100506\") " Oct 30 13:35:22.523976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.521994 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1fb92c3-28ac-4223-acee-0cc0b8100506-kube-api-access-ccqfg\" (UniqueName: \"kubernetes.io/projected/e1fb92c3-28ac-4223-acee-0cc0b8100506-kube-api-access-ccqfg\") pod \"e1fb92c3-28ac-4223-acee-0cc0b8100506\" (UID: \"e1fb92c3-28ac-4223-acee-0cc0b8100506\") " Oct 30 13:35:22.523976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.523090 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh9xf\" (UniqueName: \"kubernetes.io/projected/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c-kube-api-access-kh9xf\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:22.523976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.523802 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fb92c3-28ac-4223-acee-0cc0b8100506-plugins-config-volume" pod "e1fb92c3-28ac-4223-acee-0cc0b8100506" (UID: "e1fb92c3-28ac-4223-acee-0cc0b8100506"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:22.526779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.526735 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fb92c3-28ac-4223-acee-0cc0b8100506-kube-api-access-ccqfg" pod "e1fb92c3-28ac-4223-acee-0cc0b8100506" (UID: "e1fb92c3-28ac-4223-acee-0cc0b8100506"). InnerVolumeSpecName "kube-api-access-ccqfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:22.643450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.639993 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fb92c3-28ac-4223-acee-0cc0b8100506-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:22.643450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.640030 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccqfg\" (UniqueName: \"kubernetes.io/projected/e1fb92c3-28ac-4223-acee-0cc0b8100506-kube-api-access-ccqfg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:22.735731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:22.735605 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:23.015649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.015604 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:23.346602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.346565 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:23.367409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.367372 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7"] Oct 30 13:35:23.461920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.461865 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz" Oct 30 13:35:23.469962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.469909 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f0a31b-5cd4b9cbdb-v82d7"] Oct 30 13:35:23.565457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.556056 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g"] Oct 30 13:35:23.598106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.591442 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e1fb92c3-28ac-4223-acee-0cc0b8100506" containerName="payload-pre-processing" Oct 30 13:35:23.598106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.591476 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1fb92c3-28ac-4223-acee-0cc0b8100506" containerName="payload-pre-processing" Oct 30 13:35:23.598106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.591646 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5c5aa42c-f531-44af-9bd7-9e2d600d6a1c" containerName="payload-processing" Oct 30 13:35:23.598106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.591658 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5aa42c-f531-44af-9bd7-9e2d600d6a1c" containerName="payload-processing" Oct 30 13:35:23.598106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.592247 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1fb92c3-28ac-4223-acee-0cc0b8100506" containerName="payload-pre-processing" Oct 30 13:35:23.598106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.592297 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5aa42c-f531-44af-9bd7-9e2d600d6a1c" containerName="payload-processing" Oct 30 13:35:23.608945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.607786 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:35:23.627352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.624626 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2b254d-dockercfg-jhskb\"" Oct 30 13:35:23.627352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.624813 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2b254d\"" Oct 30 13:35:23.627352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.624862 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g"] Oct 30 13:35:23.643356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.638364 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz"] Oct 30 13:35:23.653266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.653183 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f0a31b-5dccb7995-dl4wz"] Oct 30 13:35:23.729348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.701365 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddeca63d-c87d-4204-92f1-d1972cdee454-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g\" (UID: \"ddeca63d-c87d-4204-92f1-d1972cdee454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:35:23.729348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.701442 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92wm\" (UniqueName: \"kubernetes.io/projected/ddeca63d-c87d-4204-92f1-d1972cdee454-kube-api-access-m92wm\") pod \"payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g\" (UID: \"ddeca63d-c87d-4204-92f1-d1972cdee454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:35:23.832344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.817513 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddeca63d-c87d-4204-92f1-d1972cdee454-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g\" (UID: \"ddeca63d-c87d-4204-92f1-d1972cdee454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:35:23.832344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.817575 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92wm\" (UniqueName: \"kubernetes.io/projected/ddeca63d-c87d-4204-92f1-d1972cdee454-kube-api-access-m92wm\") pod \"payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g\" (UID: \"ddeca63d-c87d-4204-92f1-d1972cdee454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:35:23.832344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.819707 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddeca63d-c87d-4204-92f1-d1972cdee454-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g\" (UID: \"ddeca63d-c87d-4204-92f1-d1972cdee454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:35:23.882486 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:23.882337 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92wm\" (UniqueName: \"kubernetes.io/projected/ddeca63d-c87d-4204-92f1-d1972cdee454-kube-api-access-m92wm\") pod \"payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g\" (UID: \"ddeca63d-c87d-4204-92f1-d1972cdee454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:35:23.920346 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:35:23.918441 5049 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fb92c3_28ac_4223_acee_0cc0b8100506.slice/crio-861715ce6c426936107ab027571029773f3ccaea1c2e331a2effafedac3377c0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fb92c3_28ac_4223_acee_0cc0b8100506.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:35:24.011774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:24.011726 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:35:24.340024 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:24.339991 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g"] Oct 30 13:35:24.341497 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:24.341471 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddeca63d_c87d_4204_92f1_d1972cdee454.slice/crio-f1540b04ebb92840ad73718127616c902609206bdc71c48b9d09b21e0f4472c2 WatchSource:0}: Error finding container f1540b04ebb92840ad73718127616c902609206bdc71c48b9d09b21e0f4472c2: Status 404 returned error can't find the container with id f1540b04ebb92840ad73718127616c902609206bdc71c48b9d09b21e0f4472c2 Oct 30 13:35:24.481947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:24.481861 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" event={"ID":"ddeca63d-c87d-4204-92f1-d1972cdee454","Type":"ContainerStarted","Data":"d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774"} Oct 30 13:35:24.482432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:24.481986 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" event={"ID":"ddeca63d-c87d-4204-92f1-d1972cdee454","Type":"ContainerStarted","Data":"f1540b04ebb92840ad73718127616c902609206bdc71c48b9d09b21e0f4472c2"} Oct 30 13:35:24.482432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:24.482269 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:35:24.522858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:24.522803 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" podStartSLOduration=1.522788464 podStartE2EDuration="1.522788464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:24.518573052 +0000 UTC m=+2631.333767475" watchObservedRunningTime="2026-10-30 13:35:24.522788464 +0000 UTC m=+2631.337982896" Oct 30 13:35:24.798459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:24.798432 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:25.002342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.000657 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:25.010379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.004345 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw"] Oct 30 13:35:25.028994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.028657 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:35:25.041021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.040989 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5857c9db-264a-4342-b143-de98a1498f02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw\" (UID: \"5857c9db-264a-4342-b143-de98a1498f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:35:25.041248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.041096 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggt85\" (UniqueName: \"kubernetes.io/projected/5857c9db-264a-4342-b143-de98a1498f02-kube-api-access-ggt85\") pod \"payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw\" (UID: \"5857c9db-264a-4342-b143-de98a1498f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:35:25.041550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.041526 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw"] Oct 30 13:35:25.044596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.044234 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cf3c79-dockercfg-mvmc7\"" Oct 30 13:35:25.044747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.044729 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cf3c79\"" Oct 30 13:35:25.142842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.142758 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggt85\" (UniqueName: \"kubernetes.io/projected/5857c9db-264a-4342-b143-de98a1498f02-kube-api-access-ggt85\") pod \"payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw\" (UID: \"5857c9db-264a-4342-b143-de98a1498f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:35:25.143050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.143032 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5857c9db-264a-4342-b143-de98a1498f02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw\" (UID: \"5857c9db-264a-4342-b143-de98a1498f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:35:25.143930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.143895 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5857c9db-264a-4342-b143-de98a1498f02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw\" (UID: \"5857c9db-264a-4342-b143-de98a1498f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:35:25.164615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.164579 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggt85\" (UniqueName: \"kubernetes.io/projected/5857c9db-264a-4342-b143-de98a1498f02-kube-api-access-ggt85\") pod \"payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw\" (UID: \"5857c9db-264a-4342-b143-de98a1498f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:35:25.338664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.338632 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5aa42c-f531-44af-9bd7-9e2d600d6a1c" path="/var/lib/kubelet/pods/5c5aa42c-f531-44af-9bd7-9e2d600d6a1c/volumes" Oct 30 13:35:25.339093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.339079 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1fb92c3-28ac-4223-acee-0cc0b8100506" path="/var/lib/kubelet/pods/e1fb92c3-28ac-4223-acee-0cc0b8100506/volumes" Oct 30 13:35:25.352276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.352248 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:35:25.691391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.690014 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:25.691391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.690473 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx"] Oct 30 13:35:25.726343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.725547 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx"] Oct 30 13:35:25.726343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.725673 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:25.754344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.747692 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:35:25.754344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.748213 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:35:25.817365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.815705 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1ba611-lhppx\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:25.817365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.815746 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbq6\" (UniqueName: \"kubernetes.io/projected/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-kube-api-access-xkbq6\") pod \"maas-api-revoke-sub-6e1ba611-lhppx\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:25.817365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.815877 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1ba611-lhppx\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:25.917666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.917625 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1ba611-lhppx\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:25.917879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.917683 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbq6\" (UniqueName: \"kubernetes.io/projected/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-kube-api-access-xkbq6\") pod \"maas-api-revoke-sub-6e1ba611-lhppx\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:25.917879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.917746 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1ba611-lhppx\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:25.918500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.918471 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1ba611-lhppx\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:25.920435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.920411 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1ba611-lhppx\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:25.940290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.940253 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbq6\" (UniqueName: \"kubernetes.io/projected/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-kube-api-access-xkbq6\") pod \"maas-api-revoke-sub-6e1ba611-lhppx\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:25.982138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:25.982108 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw"] Oct 30 13:35:25.984243 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:25.984212 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5857c9db_264a_4342_b143_de98a1498f02.slice/crio-b246ac4232d0a6342de56c8e9e7fab9554b10cf13abc5b788b20f8e33bac43ae WatchSource:0}: Error finding container b246ac4232d0a6342de56c8e9e7fab9554b10cf13abc5b788b20f8e33bac43ae: Status 404 returned error can't find the container with id b246ac4232d0a6342de56c8e9e7fab9554b10cf13abc5b788b20f8e33bac43ae Oct 30 13:35:26.069610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:26.069590 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:26.397618 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:26.397579 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f6d1e6_5f55_4ce1_9e5b_ac319ebc16d7.slice/crio-a83aacc933c17a398bd52f6032725fc268a0990bd740b076fb950fa26d80a555 WatchSource:0}: Error finding container a83aacc933c17a398bd52f6032725fc268a0990bd740b076fb950fa26d80a555: Status 404 returned error can't find the container with id a83aacc933c17a398bd52f6032725fc268a0990bd740b076fb950fa26d80a555 Oct 30 13:35:26.397739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:26.397645 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx"] Oct 30 13:35:26.632082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:26.632046 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" event={"ID":"5857c9db-264a-4342-b143-de98a1498f02","Type":"ContainerStarted","Data":"ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a"} Oct 30 13:35:26.632286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:26.632105 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" event={"ID":"5857c9db-264a-4342-b143-de98a1498f02","Type":"ContainerStarted","Data":"b246ac4232d0a6342de56c8e9e7fab9554b10cf13abc5b788b20f8e33bac43ae"} Oct 30 13:35:26.632286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:26.632146 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:35:26.633708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:26.633680 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" event={"ID":"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7","Type":"ContainerStarted","Data":"a83aacc933c17a398bd52f6032725fc268a0990bd740b076fb950fa26d80a555"} Oct 30 13:35:26.676783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:26.676678 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" podStartSLOduration=2.676661859 podStartE2EDuration="2.676661859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:26.671645497 +0000 UTC m=+2633.486839921" watchObservedRunningTime="2026-10-30 13:35:26.676661859 +0000 UTC m=+2633.491856282" Oct 30 13:35:26.683482 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:35:26.683440 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:35:27.510923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:27.510879 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:27.646147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:27.646113 5049 generic.go:356] "Generic (PLEG): container finished" podID="54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7" containerID="1b046004257156ff6307e7b67a41aab9f5c809dab15fcc86d5b8646fe4961e6e" exitCode=0 Oct 30 13:35:27.646411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:27.646202 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" event={"ID":"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7","Type":"ContainerDied","Data":"1b046004257156ff6307e7b67a41aab9f5c809dab15fcc86d5b8646fe4961e6e"} Oct 30 13:35:28.853528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:28.853500 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:28.980651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:28.980561 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-kube-api-access-xkbq6\" (UniqueName: \"kubernetes.io/projected/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-kube-api-access-xkbq6\") pod \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " Oct 30 13:35:28.980651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:28.980625 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-openshift-ca\") pod \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " Oct 30 13:35:28.980905 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:28.980824 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-xks-ca\") pod \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\" (UID: \"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7\") " Oct 30 13:35:28.981069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:28.981037 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-openshift-ca" pod "54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7" (UID: "54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:28.981547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:28.981525 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:28.982918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:28.982892 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-xks-ca" pod "54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7" (UID: "54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:28.983010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:28.982958 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-kube-api-access-xkbq6" pod "54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7" (UID: "54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7"). InnerVolumeSpecName "kube-api-access-xkbq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:29.083857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:29.083825 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:29.083857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:29.083856 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkbq6\" (UniqueName: \"kubernetes.io/projected/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7-kube-api-access-xkbq6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:29.236242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:29.236155 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:29.512822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:29.512721 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:29.676243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:29.676206 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" Oct 30 13:35:29.676469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:29.676219 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx" event={"ID":"54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7","Type":"ContainerDied","Data":"a83aacc933c17a398bd52f6032725fc268a0990bd740b076fb950fa26d80a555"} Oct 30 13:35:29.676469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:29.676406 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a83aacc933c17a398bd52f6032725fc268a0990bd740b076fb950fa26d80a555" Oct 30 13:35:30.662860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.662827 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg"] Oct 30 13:35:30.663728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.663708 5049 cpu_manager.go:414] "containerMap: removing container" podUID="54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7" containerName="revoke-keys" Oct 30 13:35:30.663728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.663728 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7" containerName="revoke-keys" Oct 30 13:35:30.663919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.663910 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7" containerName="revoke-keys" Oct 30 13:35:30.667941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.667914 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:30.692774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.692744 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:35:30.693107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.693086 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:35:30.701449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.701418 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg"] Oct 30 13:35:30.822770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.822731 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f33aa610-q9jvg\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:30.822985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.822783 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f33aa610-q9jvg\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:30.822985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.822874 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrns\" (UniqueName: \"kubernetes.io/projected/62ce81e3-250c-4960-ad10-9efa2f145e60-kube-api-access-xkrns\") pod \"maas-api-revoke-sub-f33aa610-q9jvg\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:30.925352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.925246 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f33aa610-q9jvg\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:30.925352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.925314 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f33aa610-q9jvg\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:30.925568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.925370 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrns\" (UniqueName: \"kubernetes.io/projected/62ce81e3-250c-4960-ad10-9efa2f145e60-kube-api-access-xkrns\") pod \"maas-api-revoke-sub-f33aa610-q9jvg\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:30.926143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.926111 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f33aa610-q9jvg\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:30.928005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.927979 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f33aa610-q9jvg\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:30.952546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.952518 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrns\" (UniqueName: \"kubernetes.io/projected/62ce81e3-250c-4960-ad10-9efa2f145e60-kube-api-access-xkrns\") pod \"maas-api-revoke-sub-f33aa610-q9jvg\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:30.983537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:30.983509 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:31.187307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.187219 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh"] Oct 30 13:35:31.192715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.192691 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.194265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.194237 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" podUID="5f3be1ca-961e-494a-8a13-68f3789695fd" containerName="main" containerID="cri-o://438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f" gracePeriod=45 Oct 30 13:35:31.211690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.211663 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-428a0e-serving-cert\"" Oct 30 13:35:31.246673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.243985 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh"] Oct 30 13:35:31.320614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.320583 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg"] Oct 30 13:35:31.320989 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:31.320965 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ce81e3_250c_4960_ad10_9efa2f145e60.slice/crio-62166477b020d33f95acd161a381fb785e109ac6b58a3e40f019936c1e1e925d WatchSource:0}: Error finding container 62166477b020d33f95acd161a381fb785e109ac6b58a3e40f019936c1e1e925d: Status 404 returned error can't find the container with id 62166477b020d33f95acd161a381fb785e109ac6b58a3e40f019936c1e1e925d Oct 30 13:35:31.335500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.335473 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n872h\" (UniqueName: \"kubernetes.io/projected/b8c15b62-e6fc-4b63-956b-41c266d073dc-kube-api-access-n872h\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.335803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.335781 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.336064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.336049 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-metrics-certs\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.336198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.336184 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c15b62-e6fc-4b63-956b-41c266d073dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.342215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.342159 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:31.437685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.437602 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.437878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.437856 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-metrics-certs\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.437969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.437951 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c15b62-e6fc-4b63-956b-41c266d073dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.439243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.439210 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c15b62-e6fc-4b63-956b-41c266d073dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.439417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.439263 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n872h\" (UniqueName: \"kubernetes.io/projected/b8c15b62-e6fc-4b63-956b-41c266d073dc-kube-api-access-n872h\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.440935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.440909 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.441372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.441352 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-metrics-certs\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.504371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.501482 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n872h\" (UniqueName: \"kubernetes.io/projected/b8c15b62-e6fc-4b63-956b-41c266d073dc-kube-api-access-n872h\") pod \"maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.531086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.514458 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:31.563649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.563606 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:31.573831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.573803 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:35:31.650529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.649250 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f3be1ca-961e-494a-8a13-68f3789695fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3be1ca-961e-494a-8a13-68f3789695fd-tls-certs\") pod \"5f3be1ca-961e-494a-8a13-68f3789695fd\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " Oct 30 13:35:31.650529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.649449 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-tmp-dir\") pod \"5f3be1ca-961e-494a-8a13-68f3789695fd\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " Oct 30 13:35:31.650529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.649527 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-model-cache\") pod \"5f3be1ca-961e-494a-8a13-68f3789695fd\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " Oct 30 13:35:31.650529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.649578 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-home\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-home\") pod \"5f3be1ca-961e-494a-8a13-68f3789695fd\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " Oct 30 13:35:31.650529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.649665 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3be1ca-961e-494a-8a13-68f3789695fd-kube-api-access-gljk6\" (UniqueName: \"kubernetes.io/projected/5f3be1ca-961e-494a-8a13-68f3789695fd-kube-api-access-gljk6\") pod \"5f3be1ca-961e-494a-8a13-68f3789695fd\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " Oct 30 13:35:31.650529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.649697 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-dshm\") pod \"5f3be1ca-961e-494a-8a13-68f3789695fd\" (UID: \"5f3be1ca-961e-494a-8a13-68f3789695fd\") " Oct 30 13:35:31.653029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.652754 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-dshm" pod "5f3be1ca-961e-494a-8a13-68f3789695fd" (UID: "5f3be1ca-961e-494a-8a13-68f3789695fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:31.653398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.653373 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-model-cache" pod "5f3be1ca-961e-494a-8a13-68f3789695fd" (UID: "5f3be1ca-961e-494a-8a13-68f3789695fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:31.654265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.653571 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-tmp-dir" pod "5f3be1ca-961e-494a-8a13-68f3789695fd" (UID: "5f3be1ca-961e-494a-8a13-68f3789695fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:31.654265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.653761 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-home" pod "5f3be1ca-961e-494a-8a13-68f3789695fd" (UID: "5f3be1ca-961e-494a-8a13-68f3789695fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:31.655074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.655035 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3be1ca-961e-494a-8a13-68f3789695fd-tls-certs" pod "5f3be1ca-961e-494a-8a13-68f3789695fd" (UID: "5f3be1ca-961e-494a-8a13-68f3789695fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:31.656444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.656422 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3be1ca-961e-494a-8a13-68f3789695fd-kube-api-access-gljk6" pod "5f3be1ca-961e-494a-8a13-68f3789695fd" (UID: "5f3be1ca-961e-494a-8a13-68f3789695fd"). InnerVolumeSpecName "kube-api-access-gljk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:31.705272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.704874 5049 generic.go:356] "Generic (PLEG): container finished" podID="5f3be1ca-961e-494a-8a13-68f3789695fd" containerID="438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f" exitCode=0 Oct 30 13:35:31.705272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.704922 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" Oct 30 13:35:31.705272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.705023 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" event={"ID":"5f3be1ca-961e-494a-8a13-68f3789695fd","Type":"ContainerDied","Data":"438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f"} Oct 30 13:35:31.705272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.705054 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf" event={"ID":"5f3be1ca-961e-494a-8a13-68f3789695fd","Type":"ContainerDied","Data":"eeb78c965106f4a18df32d941da93df956e48ad32184f4f10bc73bc59e0b5898"} Oct 30 13:35:31.705272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.705078 5049 scope.go:122] "RemoveContainer" containerID="438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f" Oct 30 13:35:31.711514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.711396 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" event={"ID":"62ce81e3-250c-4960-ad10-9efa2f145e60","Type":"ContainerStarted","Data":"62166477b020d33f95acd161a381fb785e109ac6b58a3e40f019936c1e1e925d"} Oct 30 13:35:31.725014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.724991 5049 scope.go:122] "RemoveContainer" containerID="438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f" Oct 30 13:35:31.725386 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:35:31.725363 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f\": container with ID starting with 438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f not found: ID does not exist" containerID="438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f" Oct 30 13:35:31.725453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.725395 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f"} err="failed to get container status \"438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f\": rpc error: code = NotFound desc = could not find container \"438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f\": container with ID starting with 438d2a2dd26d8820388a8c2264358be42542c9fefb1fdf6410e4372dfc32bf4f not found: ID does not exist" Oct 30 13:35:31.751996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.751961 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3be1ca-961e-494a-8a13-68f3789695fd-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.751996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.751991 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.751996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.752000 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.752293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.752010 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.752293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.752022 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gljk6\" (UniqueName: \"kubernetes.io/projected/5f3be1ca-961e-494a-8a13-68f3789695fd-kube-api-access-gljk6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.752293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.752031 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3be1ca-961e-494a-8a13-68f3789695fd-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.768437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.768404 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf"] Oct 30 13:35:31.792345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.785858 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6da60a/auth-test-model-6da60a-kserve-79d4bd6885-qhmtf"] Oct 30 13:35:31.877240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.877212 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh"] Oct 30 13:35:31.879484 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:31.879456 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c15b62_e6fc_4b63_956b_41c266d073dc.slice/crio-37d3263688d06605eef415e24f938a5bfe4d2c7b4de496cd51aebe21d87ce1c0 WatchSource:0}: Error finding container 37d3263688d06605eef415e24f938a5bfe4d2c7b4de496cd51aebe21d87ce1c0: Status 404 returned error can't find the container with id 37d3263688d06605eef415e24f938a5bfe4d2c7b4de496cd51aebe21d87ce1c0 Oct 30 13:35:31.885666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:31.885643 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:35:32.771347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:32.770455 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" event={"ID":"b8c15b62-e6fc-4b63-956b-41c266d073dc","Type":"ContainerStarted","Data":"dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91"} Oct 30 13:35:32.771347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:32.770525 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" event={"ID":"b8c15b62-e6fc-4b63-956b-41c266d073dc","Type":"ContainerStarted","Data":"37d3263688d06605eef415e24f938a5bfe4d2c7b4de496cd51aebe21d87ce1c0"} Oct 30 13:35:32.789341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:32.786205 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:32.811363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:32.811023 5049 generic.go:356] "Generic (PLEG): container finished" podID="62ce81e3-250c-4960-ad10-9efa2f145e60" containerID="3d349c9e7b75cc01055a223c40fe2953229c3bb3a1e655eb7eb9352951d7f420" exitCode=0 Oct 30 13:35:32.811363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:32.811054 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" event={"ID":"62ce81e3-250c-4960-ad10-9efa2f145e60","Type":"ContainerDied","Data":"3d349c9e7b75cc01055a223c40fe2953229c3bb3a1e655eb7eb9352951d7f420"} Oct 30 13:35:32.836695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:32.836637 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" podStartSLOduration=1.468495199 podStartE2EDuration="1.836622878s" totalImagesPullingTime="368.127679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:32.832541921 +0000 UTC m=+2639.647736356" watchObservedRunningTime="2026-10-30 13:35:32.836622878 +0000 UTC m=+2639.651817354" Oct 30 13:35:33.204275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.204244 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w"] Oct 30 13:35:33.205341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.205300 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5f3be1ca-961e-494a-8a13-68f3789695fd" containerName="main" Oct 30 13:35:33.205341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.205342 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3be1ca-961e-494a-8a13-68f3789695fd" containerName="main" Oct 30 13:35:33.205550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.205496 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3be1ca-961e-494a-8a13-68f3789695fd" containerName="main" Oct 30 13:35:33.209733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.209711 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.226238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.226210 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b4fb25-serving-cert\"" Oct 30 13:35:33.234746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.234700 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w"] Oct 30 13:35:33.317163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.317110 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-metrics-certs\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.317394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.317266 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.317394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.317367 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtww\" (UniqueName: \"kubernetes.io/projected/fbcea945-63c3-4aae-8613-65944877613a-kube-api-access-djtww\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.317530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.317409 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcea945-63c3-4aae-8613-65944877613a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.338268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.338216 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:33.345101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.345070 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3be1ca-961e-494a-8a13-68f3789695fd" path="/var/lib/kubelet/pods/5f3be1ca-961e-494a-8a13-68f3789695fd/volumes" Oct 30 13:35:33.435348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.425691 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-metrics-certs\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.435348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.425873 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.435348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.425933 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtww\" (UniqueName: \"kubernetes.io/projected/fbcea945-63c3-4aae-8613-65944877613a-kube-api-access-djtww\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.435348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.425975 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcea945-63c3-4aae-8613-65944877613a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.435348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.427059 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcea945-63c3-4aae-8613-65944877613a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.439342 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:35:33.437541 5049 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-serving-cert: secret "maas-api-e2e-ipp-a-b4fb25-serving-cert" not found Oct 30 13:35:33.439342 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:35:33.437633 5049 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-maas-api-tls podName:fbcea945-63c3-4aae-8613-65944877613a nodeName:}" failed. No retries permitted until 2026-10-30 13:35:33.937609544 +0000 UTC m=+2640.752803959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-maas-api-tls") pod "maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" (UID: "fbcea945-63c3-4aae-8613-65944877613a") : secret "maas-api-e2e-ipp-a-b4fb25-serving-cert" not found Oct 30 13:35:33.496056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.493468 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtww\" (UniqueName: \"kubernetes.io/projected/fbcea945-63c3-4aae-8613-65944877613a-kube-api-access-djtww\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.499962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.499930 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:33.509821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.509785 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-metrics-certs\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:33.694348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:33.694295 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:34.036798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.036764 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:34.039641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.039591 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:34.065822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.065792 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:34.161192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.161103 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:34.263338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.263285 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-openshift-ca\") pod \"62ce81e3-250c-4960-ad10-9efa2f145e60\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " Oct 30 13:35:34.263502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.263456 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ce81e3-250c-4960-ad10-9efa2f145e60-kube-api-access-xkrns\" (UniqueName: \"kubernetes.io/projected/62ce81e3-250c-4960-ad10-9efa2f145e60-kube-api-access-xkrns\") pod \"62ce81e3-250c-4960-ad10-9efa2f145e60\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " Oct 30 13:35:34.263596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.263577 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-xks-ca\") pod \"62ce81e3-250c-4960-ad10-9efa2f145e60\" (UID: \"62ce81e3-250c-4960-ad10-9efa2f145e60\") " Oct 30 13:35:34.265638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.265611 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-openshift-ca" pod "62ce81e3-250c-4960-ad10-9efa2f145e60" (UID: "62ce81e3-250c-4960-ad10-9efa2f145e60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:34.282338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.273471 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-xks-ca" pod "62ce81e3-250c-4960-ad10-9efa2f145e60" (UID: "62ce81e3-250c-4960-ad10-9efa2f145e60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:34.282338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.273872 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ce81e3-250c-4960-ad10-9efa2f145e60-kube-api-access-xkrns" pod "62ce81e3-250c-4960-ad10-9efa2f145e60" (UID: "62ce81e3-250c-4960-ad10-9efa2f145e60"). InnerVolumeSpecName "kube-api-access-xkrns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:34.366728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.366690 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:34.366728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.366728 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce81e3-250c-4960-ad10-9efa2f145e60-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:34.367011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.366743 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkrns\" (UniqueName: \"kubernetes.io/projected/62ce81e3-250c-4960-ad10-9efa2f145e60-kube-api-access-xkrns\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:34.509387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.508859 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w"] Oct 30 13:35:34.509387 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:34.509200 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcea945_63c3_4aae_8613_65944877613a.slice/crio-d8822fa03942c03577f9d7ae11030ca3f257fe854f9c37277d60ffeb47a2c610 WatchSource:0}: Error finding container d8822fa03942c03577f9d7ae11030ca3f257fe854f9c37277d60ffeb47a2c610: Status 404 returned error can't find the container with id d8822fa03942c03577f9d7ae11030ca3f257fe854f9c37277d60ffeb47a2c610 Oct 30 13:35:34.837895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.837868 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" Oct 30 13:35:34.838056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.837868 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg" event={"ID":"62ce81e3-250c-4960-ad10-9efa2f145e60","Type":"ContainerDied","Data":"62166477b020d33f95acd161a381fb785e109ac6b58a3e40f019936c1e1e925d"} Oct 30 13:35:34.838056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.838044 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62166477b020d33f95acd161a381fb785e109ac6b58a3e40f019936c1e1e925d" Oct 30 13:35:34.840700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:34.840674 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" event={"ID":"fbcea945-63c3-4aae-8613-65944877613a","Type":"ContainerStarted","Data":"d8822fa03942c03577f9d7ae11030ca3f257fe854f9c37277d60ffeb47a2c610"} Oct 30 13:35:35.335744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.335446 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:35.626574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.626496 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:35:35.852764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.852731 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" event={"ID":"fbcea945-63c3-4aae-8613-65944877613a","Type":"ContainerStarted","Data":"b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78"} Oct 30 13:35:35.852961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.852837 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:35.897021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.896917 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" podStartSLOduration=2.5873288519999997 podStartE2EDuration="2.896903307s" totalImagesPullingTime="309.574455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:35.892196608 +0000 UTC m=+2642.707391030" watchObservedRunningTime="2026-10-30 13:35:35.896903307 +0000 UTC m=+2642.712097730" Oct 30 13:35:35.914865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.914835 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9"] Oct 30 13:35:35.915756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.915739 5049 cpu_manager.go:414] "containerMap: removing container" podUID="62ce81e3-250c-4960-ad10-9efa2f145e60" containerName="revoke-keys" Oct 30 13:35:35.915815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.915757 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ce81e3-250c-4960-ad10-9efa2f145e60" containerName="revoke-keys" Oct 30 13:35:35.915910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.915901 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ce81e3-250c-4960-ad10-9efa2f145e60" containerName="revoke-keys" Oct 30 13:35:35.919997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.919974 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:35.937269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.937210 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:35:35.937429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.937416 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:35:35.965305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.965272 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9"] Oct 30 13:35:35.994916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.994880 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aef9da70-g22n9\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:35.995224 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.995197 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aef9da70-g22n9\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:35.995284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:35.995268 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m444\" (UniqueName: \"kubernetes.io/projected/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-kube-api-access-8m444\") pod \"maas-api-revoke-sub-aef9da70-g22n9\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:36.096935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.096896 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aef9da70-g22n9\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:36.097132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.097117 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aef9da70-g22n9\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:36.097181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.097161 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m444\" (UniqueName: \"kubernetes.io/projected/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-kube-api-access-8m444\") pod \"maas-api-revoke-sub-aef9da70-g22n9\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:36.097832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.097807 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aef9da70-g22n9\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:36.099463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.099439 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aef9da70-g22n9\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:36.119147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.119119 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m444\" (UniqueName: \"kubernetes.io/projected/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-kube-api-access-8m444\") pod \"maas-api-revoke-sub-aef9da70-g22n9\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:36.238065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.237975 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:36.561833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.561799 5049 scope.go:122] "RemoveContainer" containerID="d352c6b78996abe7227f2056d70157df4ac7d03a6d6f0fe42f3752f3831596d1" Oct 30 13:35:36.570576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.570542 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9"] Oct 30 13:35:36.579526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.579503 5049 scope.go:122] "RemoveContainer" containerID="fbb41365b1db700892acdf302872986c99980bef4a0ec80a6fcaae2ee5751948" Oct 30 13:35:36.581587 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:36.581563 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8aaedb_2406_4287_9faa_6a7f261b3cc5.slice/crio-2b2d7d12af856c160d26e3a5749adcbeafb66212c6b254e7bb560f46ae721dc0 WatchSource:0}: Error finding container 2b2d7d12af856c160d26e3a5749adcbeafb66212c6b254e7bb560f46ae721dc0: Status 404 returned error can't find the container with id 2b2d7d12af856c160d26e3a5749adcbeafb66212c6b254e7bb560f46ae721dc0 Oct 30 13:35:36.603052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.603025 5049 scope.go:122] "RemoveContainer" containerID="b327057ff96e44768a4c7250c097f59aa7ae42bbca1e5a5ae7450e7cfdce97bf" Oct 30 13:35:36.863478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:36.863388 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" event={"ID":"0a8aaedb-2406-4287-9faa-6a7f261b3cc5","Type":"ContainerStarted","Data":"2b2d7d12af856c160d26e3a5749adcbeafb66212c6b254e7bb560f46ae721dc0"} Oct 30 13:35:37.140426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:37.136554 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:37.650204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:37.650174 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:35:37.874269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:37.874233 5049 generic.go:356] "Generic (PLEG): container finished" podID="0a8aaedb-2406-4287-9faa-6a7f261b3cc5" containerID="a05d9a68dea6d260835acf0dab56c6c1d123d7589decac4f616cfbf963dac186" exitCode=0 Oct 30 13:35:37.874549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:37.874286 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" event={"ID":"0a8aaedb-2406-4287-9faa-6a7f261b3cc5","Type":"ContainerDied","Data":"a05d9a68dea6d260835acf0dab56c6c1d123d7589decac4f616cfbf963dac186"} Oct 30 13:35:38.529230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.529194 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4"] Oct 30 13:35:38.529621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.529518 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" podUID="fff531b7-9d8c-4fa0-9cc8-744df2fc599c" containerName="maas-api" containerID="cri-o://b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36" gracePeriod=30 Oct 30 13:35:38.821272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.821242 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:38.841312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.841286 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:35:38.955348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.954752 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-maas-api-tls\") pod \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " Oct 30 13:35:38.955348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.954819 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-kube-api-access-lddpg\" (UniqueName: \"kubernetes.io/projected/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-kube-api-access-lddpg\") pod \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " Oct 30 13:35:38.955348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.955095 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-metrics-certs\") pod \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " Oct 30 13:35:38.955348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.955145 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-openshift-service-ca\") pod \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\" (UID: \"fff531b7-9d8c-4fa0-9cc8-744df2fc599c\") " Oct 30 13:35:38.970625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.969353 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-openshift-service-ca" pod "fff531b7-9d8c-4fa0-9cc8-744df2fc599c" (UID: "fff531b7-9d8c-4fa0-9cc8-744df2fc599c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:38.980229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.980186 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:38.980560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.980517 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-maas-api-tls" pod "fff531b7-9d8c-4fa0-9cc8-744df2fc599c" (UID: "fff531b7-9d8c-4fa0-9cc8-744df2fc599c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:38.982407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.982353 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-kube-api-access-lddpg" pod "fff531b7-9d8c-4fa0-9cc8-744df2fc599c" (UID: "fff531b7-9d8c-4fa0-9cc8-744df2fc599c"). InnerVolumeSpecName "kube-api-access-lddpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:38.982537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.982457 5049 generic.go:356] "Generic (PLEG): container finished" podID="fff531b7-9d8c-4fa0-9cc8-744df2fc599c" containerID="b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36" exitCode=0 Oct 30 13:35:38.983136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.982773 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" Oct 30 13:35:38.983136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.982838 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" event={"ID":"fff531b7-9d8c-4fa0-9cc8-744df2fc599c","Type":"ContainerDied","Data":"b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36"} Oct 30 13:35:38.983136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.982887 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4" event={"ID":"fff531b7-9d8c-4fa0-9cc8-744df2fc599c","Type":"ContainerDied","Data":"3ec161d4e0e56281df362d3aaeb8a91c692076a3538169a25a9b585dafc0e52c"} Oct 30 13:35:38.983136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:38.982914 5049 scope.go:122] "RemoveContainer" containerID="b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36" Oct 30 13:35:39.025365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.023463 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-metrics-certs" pod "fff531b7-9d8c-4fa0-9cc8-744df2fc599c" (UID: "fff531b7-9d8c-4fa0-9cc8-744df2fc599c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:39.036155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.036125 5049 scope.go:122] "RemoveContainer" containerID="b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36" Oct 30 13:35:39.036600 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:35:39.036571 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36\": container with ID starting with b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36 not found: ID does not exist" containerID="b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36" Oct 30 13:35:39.036731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.036612 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36"} err="failed to get container status \"b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36\": rpc error: code = NotFound desc = could not find container \"b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36\": container with ID starting with b2bc43c7ac4da8f01e0d7adc0a3dc996c02a37b4e15088a0123150a645d16c36 not found: ID does not exist" Oct 30 13:35:39.061517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.059824 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk"] Oct 30 13:35:39.068127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.068023 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275ab4bf-vnzdk"] Oct 30 13:35:39.089767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.087963 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lddpg\" (UniqueName: \"kubernetes.io/projected/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-kube-api-access-lddpg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:39.089767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.087999 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:39.089767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.088014 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff531b7-9d8c-4fa0-9cc8-744df2fc599c-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:39.329395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.328296 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:39.346840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.346455 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-kube-api-access-8m444\" (UniqueName: \"kubernetes.io/projected/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-kube-api-access-8m444\") pod \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " Oct 30 13:35:39.346840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.346499 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-xks-ca\") pod \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " Oct 30 13:35:39.346840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.346667 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-openshift-ca\") pod \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\" (UID: \"0a8aaedb-2406-4287-9faa-6a7f261b3cc5\") " Oct 30 13:35:39.350488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.348582 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-openshift-ca" pod "0a8aaedb-2406-4287-9faa-6a7f261b3cc5" (UID: "0a8aaedb-2406-4287-9faa-6a7f261b3cc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:39.364553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.364514 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-kube-api-access-8m444" pod "0a8aaedb-2406-4287-9faa-6a7f261b3cc5" (UID: "0a8aaedb-2406-4287-9faa-6a7f261b3cc5"). InnerVolumeSpecName "kube-api-access-8m444". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:39.366710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.366683 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-xks-ca" pod "0a8aaedb-2406-4287-9faa-6a7f261b3cc5" (UID: "0a8aaedb-2406-4287-9faa-6a7f261b3cc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:39.368652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.368625 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47006709-da8c-4ef7-9eaf-e8de0a6ecdda" path="/var/lib/kubelet/pods/47006709-da8c-4ef7-9eaf-e8de0a6ecdda/volumes" Oct 30 13:35:39.369748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.369723 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4"] Oct 30 13:35:39.376618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.376588 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f0a31b-7847747b5b-n9mx4"] Oct 30 13:35:39.448707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.448673 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m444\" (UniqueName: \"kubernetes.io/projected/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-kube-api-access-8m444\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:39.448707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.448706 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:39.448917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.448719 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8aaedb-2406-4287-9faa-6a7f261b3cc5-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:39.571101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.571071 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:39.668133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.668042 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6"] Oct 30 13:35:39.673125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.673094 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f0a31b-qnlv6"] Oct 30 13:35:39.736252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.736218 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:39.847470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.847440 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:35:39.994474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.994400 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" Oct 30 13:35:39.994474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.994437 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9" event={"ID":"0a8aaedb-2406-4287-9faa-6a7f261b3cc5","Type":"ContainerDied","Data":"2b2d7d12af856c160d26e3a5749adcbeafb66212c6b254e7bb560f46ae721dc0"} Oct 30 13:35:39.994694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:39.994493 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b2d7d12af856c160d26e3a5749adcbeafb66212c6b254e7bb560f46ae721dc0" Oct 30 13:35:40.200626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:40.200596 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:41.013490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.013452 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l"] Oct 30 13:35:41.014363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.014344 5049 cpu_manager.go:414] "containerMap: removing container" podUID="fff531b7-9d8c-4fa0-9cc8-744df2fc599c" containerName="maas-api" Oct 30 13:35:41.014363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.014361 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff531b7-9d8c-4fa0-9cc8-744df2fc599c" containerName="maas-api" Oct 30 13:35:41.014557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.014378 5049 cpu_manager.go:414] "containerMap: removing container" podUID="0a8aaedb-2406-4287-9faa-6a7f261b3cc5" containerName="revoke-keys" Oct 30 13:35:41.014557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.014383 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8aaedb-2406-4287-9faa-6a7f261b3cc5" containerName="revoke-keys" Oct 30 13:35:41.014557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.014525 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff531b7-9d8c-4fa0-9cc8-744df2fc599c" containerName="maas-api" Oct 30 13:35:41.014557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.014537 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8aaedb-2406-4287-9faa-6a7f261b3cc5" containerName="revoke-keys" Oct 30 13:35:41.018924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.018902 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.036079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.036025 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:35:41.036228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.036090 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:35:41.042765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.042739 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l"] Oct 30 13:35:41.074827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.074784 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgpv\" (UniqueName: \"kubernetes.io/projected/ef8ae58c-a49e-46ec-9231-21e6fa097adb-kube-api-access-zlgpv\") pod \"maas-api-revoke-sub-fa3bb89b-hsh7l\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.074996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.074971 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3bb89b-hsh7l\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.075049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.074998 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3bb89b-hsh7l\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.177025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.176988 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgpv\" (UniqueName: \"kubernetes.io/projected/ef8ae58c-a49e-46ec-9231-21e6fa097adb-kube-api-access-zlgpv\") pod \"maas-api-revoke-sub-fa3bb89b-hsh7l\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.177219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.177104 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3bb89b-hsh7l\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.177219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.177124 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3bb89b-hsh7l\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.177780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.177757 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3bb89b-hsh7l\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.179679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.179652 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3bb89b-hsh7l\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.198685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.198655 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgpv\" (UniqueName: \"kubernetes.io/projected/ef8ae58c-a49e-46ec-9231-21e6fa097adb-kube-api-access-zlgpv\") pod \"maas-api-revoke-sub-fa3bb89b-hsh7l\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.333951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.333914 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:41.343705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.343670 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f490aa-577a-47dd-847f-97739ad03531" path="/var/lib/kubelet/pods/c8f490aa-577a-47dd-847f-97739ad03531/volumes" Oct 30 13:35:41.344296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.344272 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff531b7-9d8c-4fa0-9cc8-744df2fc599c" path="/var/lib/kubelet/pods/fff531b7-9d8c-4fa0-9cc8-744df2fc599c/volumes" Oct 30 13:35:41.650375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.650348 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l"] Oct 30 13:35:41.653085 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:41.653050 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8ae58c_a49e_46ec_9231_21e6fa097adb.slice/crio-dd95cfc00f710dcf17d5580c17909a2a9143be854be22eed6802b2c06a43a9c8 WatchSource:0}: Error finding container dd95cfc00f710dcf17d5580c17909a2a9143be854be22eed6802b2c06a43a9c8: Status 404 returned error can't find the container with id dd95cfc00f710dcf17d5580c17909a2a9143be854be22eed6802b2c06a43a9c8 Oct 30 13:35:41.798415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.797982 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:41.869799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:41.869772 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:35:42.020748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:42.020713 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" event={"ID":"ef8ae58c-a49e-46ec-9231-21e6fa097adb","Type":"ContainerStarted","Data":"dd95cfc00f710dcf17d5580c17909a2a9143be854be22eed6802b2c06a43a9c8"} Oct 30 13:35:43.033069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:43.032983 5049 generic.go:356] "Generic (PLEG): container finished" podID="ef8ae58c-a49e-46ec-9231-21e6fa097adb" containerID="e6d98474fb97222ede8acc84609a45f3879bee804e8ce4dd4410cdb756193c50" exitCode=0 Oct 30 13:35:43.033509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:43.033067 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" event={"ID":"ef8ae58c-a49e-46ec-9231-21e6fa097adb","Type":"ContainerDied","Data":"e6d98474fb97222ede8acc84609a45f3879bee804e8ce4dd4410cdb756193c50"} Oct 30 13:35:43.585750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:43.585706 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:44.254722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:44.254688 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:44.330954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:44.330856 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8ae58c-a49e-46ec-9231-21e6fa097adb-kube-api-access-zlgpv\" (UniqueName: \"kubernetes.io/projected/ef8ae58c-a49e-46ec-9231-21e6fa097adb-kube-api-access-zlgpv\") pod \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " Oct 30 13:35:44.331155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:44.331133 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-openshift-ca\") pod \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " Oct 30 13:35:44.331398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:44.331377 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-xks-ca\") pod \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\" (UID: \"ef8ae58c-a49e-46ec-9231-21e6fa097adb\") " Oct 30 13:35:44.331703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:44.331677 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-openshift-ca" pod "ef8ae58c-a49e-46ec-9231-21e6fa097adb" (UID: "ef8ae58c-a49e-46ec-9231-21e6fa097adb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:44.332234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:44.332213 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:44.333268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:44.333242 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8ae58c-a49e-46ec-9231-21e6fa097adb-kube-api-access-zlgpv" pod "ef8ae58c-a49e-46ec-9231-21e6fa097adb" (UID: "ef8ae58c-a49e-46ec-9231-21e6fa097adb"). InnerVolumeSpecName "kube-api-access-zlgpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:44.333956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:44.333934 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-xks-ca" pod "ef8ae58c-a49e-46ec-9231-21e6fa097adb" (UID: "ef8ae58c-a49e-46ec-9231-21e6fa097adb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:44.434351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:44.434286 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8ae58c-a49e-46ec-9231-21e6fa097adb-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:44.434351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:44.434335 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlgpv\" (UniqueName: \"kubernetes.io/projected/ef8ae58c-a49e-46ec-9231-21e6fa097adb-kube-api-access-zlgpv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:45.056724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:45.056692 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" Oct 30 13:35:45.056997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:45.056742 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l" event={"ID":"ef8ae58c-a49e-46ec-9231-21e6fa097adb","Type":"ContainerDied","Data":"dd95cfc00f710dcf17d5580c17909a2a9143be854be22eed6802b2c06a43a9c8"} Oct 30 13:35:45.056997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:45.056808 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd95cfc00f710dcf17d5580c17909a2a9143be854be22eed6802b2c06a43a9c8" Oct 30 13:35:45.407233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:45.407136 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:47.237537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:47.237502 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:47.529796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:47.529708 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:47.644245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:47.644212 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf"] Oct 30 13:35:47.644471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:47.644446 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" containerID="cri-o://9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555" gracePeriod=30 Oct 30 13:35:47.804105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:47.804069 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:48.152880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.152808 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s"] Oct 30 13:35:48.153736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.153718 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ef8ae58c-a49e-46ec-9231-21e6fa097adb" containerName="revoke-keys" Oct 30 13:35:48.153786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.153737 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8ae58c-a49e-46ec-9231-21e6fa097adb" containerName="revoke-keys" Oct 30 13:35:48.153889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.153879 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8ae58c-a49e-46ec-9231-21e6fa097adb" containerName="revoke-keys" Oct 30 13:35:48.159036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.159016 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.176832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.176808 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:35:48.176988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.176969 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:35:48.183191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.183162 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s"] Oct 30 13:35:48.318256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.318222 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.318656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.318277 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4gd\" (UniqueName: \"kubernetes.io/projected/9463d6b5-375b-416e-af3e-ecb51160691f-kube-api-access-fz4gd\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.318656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.318470 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.423394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.421079 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.423394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.421154 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4gd\" (UniqueName: \"kubernetes.io/projected/9463d6b5-375b-416e-af3e-ecb51160691f-kube-api-access-fz4gd\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.423394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.421351 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.425052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.425022 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.479900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.466827 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.479900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.468913 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4gd\" (UniqueName: \"kubernetes.io/projected/9463d6b5-375b-416e-af3e-ecb51160691f-kube-api-access-fz4gd\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.481651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.481617 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:48.626379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.623458 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc"] Oct 30 13:35:48.633688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.632765 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.656435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.656408 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc"] Oct 30 13:35:48.735185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.735099 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.735185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.735151 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88af2245-014a-4598-a442-7053d9d1e928-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.735398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.735218 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88af2245-014a-4598-a442-7053d9d1e928-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.735398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.735329 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2kr\" (UniqueName: \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-kube-api-access-mq2kr\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.735471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.735429 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.735507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.735476 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.735571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.735554 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.735620 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.735584 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.735670 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.735638 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.837725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.837687 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.837924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.837831 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.837924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.837865 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88af2245-014a-4598-a442-7053d9d1e928-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.837924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.837885 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88af2245-014a-4598-a442-7053d9d1e928-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.838100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.838067 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2kr\" (UniqueName: \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-kube-api-access-mq2kr\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.838186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.838166 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.838264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.838248 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.838347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.838303 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.838409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.838334 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.838409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.838350 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.838901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.838875 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88af2245-014a-4598-a442-7053d9d1e928-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.839080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.839055 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.839150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.839126 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.839150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.839134 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.840708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.840679 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88af2245-014a-4598-a442-7053d9d1e928-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.840811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.840796 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.861293 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:48.861269 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9463d6b5_375b_416e_af3e_ecb51160691f.slice/crio-6e3c0ea7296f82f5fc714f07c90917e989242cf2ad4d3bcf78ff495a96b2eca5 WatchSource:0}: Error finding container 6e3c0ea7296f82f5fc714f07c90917e989242cf2ad4d3bcf78ff495a96b2eca5: Status 404 returned error can't find the container with id 6e3c0ea7296f82f5fc714f07c90917e989242cf2ad4d3bcf78ff495a96b2eca5 Oct 30 13:35:48.861987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.861962 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s"] Oct 30 13:35:48.866621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.866604 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.866797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.866778 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2kr\" (UniqueName: \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-kube-api-access-mq2kr\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:48.958387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:48.958359 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:49.042465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:49.042282 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq"] Oct 30 13:35:49.050632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:49.050582 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12826b2b-lq4cq"] Oct 30 13:35:49.100016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:49.099977 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" event={"ID":"9463d6b5-375b-416e-af3e-ecb51160691f","Type":"ContainerStarted","Data":"6e3c0ea7296f82f5fc714f07c90917e989242cf2ad4d3bcf78ff495a96b2eca5"} Oct 30 13:35:49.288343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:49.288254 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc"] Oct 30 13:35:49.292721 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:35:49.292693 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88af2245_014a_4598_a442_7053d9d1e928.slice/crio-70974189a678c2c31355a80814abcc9ae62acdc2883dc3fe7fdfd278986bc98f WatchSource:0}: Error finding container 70974189a678c2c31355a80814abcc9ae62acdc2883dc3fe7fdfd278986bc98f: Status 404 returned error can't find the container with id 70974189a678c2c31355a80814abcc9ae62acdc2883dc3fe7fdfd278986bc98f Oct 30 13:35:49.295876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:49.295844 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:49.295944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:49.295912 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:49.295987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:49.295954 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:49.339648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:49.339625 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f38bfbd-51c0-40e2-aad4-6feaf8df3a95" path="/var/lib/kubelet/pods/0f38bfbd-51c0-40e2-aad4-6feaf8df3a95/volumes" Oct 30 13:35:50.134310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:50.134274 5049 generic.go:356] "Generic (PLEG): container finished" podID="9463d6b5-375b-416e-af3e-ecb51160691f" containerID="14cffb3ce43bac2f65bfaa9470b7b3acef7d41c449437c7da99e87a72379533a" exitCode=0 Oct 30 13:35:50.135302 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:50.135268 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" event={"ID":"9463d6b5-375b-416e-af3e-ecb51160691f","Type":"ContainerDied","Data":"14cffb3ce43bac2f65bfaa9470b7b3acef7d41c449437c7da99e87a72379533a"} Oct 30 13:35:50.138505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:50.138476 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" event={"ID":"88af2245-014a-4598-a442-7053d9d1e928","Type":"ContainerStarted","Data":"5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3"} Oct 30 13:35:50.138711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:50.138691 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" event={"ID":"88af2245-014a-4598-a442-7053d9d1e928","Type":"ContainerStarted","Data":"70974189a678c2c31355a80814abcc9ae62acdc2883dc3fe7fdfd278986bc98f"} Oct 30 13:35:50.242957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:50.242834 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" podStartSLOduration=2.242812148 podStartE2EDuration="2.242812148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:50.222656359 +0000 UTC m=+2657.037850783" watchObservedRunningTime="2026-10-30 13:35:50.242812148 +0000 UTC m=+2657.058006573" Oct 30 13:35:50.322895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:50.322861 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:50.958635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:50.958598 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:50.960686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:50.960653 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 30 13:35:50.960869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:50.960707 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" podUID="88af2245-014a-4598-a442-7053d9d1e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 30 13:35:51.273250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.272439 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr"] Oct 30 13:35:51.461357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.461144 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:51.603470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.603443 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:51.689589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.689562 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9463d6b5-375b-416e-af3e-ecb51160691f-kube-api-access-fz4gd\" (UniqueName: \"kubernetes.io/projected/9463d6b5-375b-416e-af3e-ecb51160691f-kube-api-access-fz4gd\") pod \"9463d6b5-375b-416e-af3e-ecb51160691f\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " Oct 30 13:35:51.690285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.690263 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-xks-ca\") pod \"9463d6b5-375b-416e-af3e-ecb51160691f\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " Oct 30 13:35:51.690502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.690481 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-openshift-ca\") pod \"9463d6b5-375b-416e-af3e-ecb51160691f\" (UID: \"9463d6b5-375b-416e-af3e-ecb51160691f\") " Oct 30 13:35:51.690975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.690948 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-openshift-ca" pod "9463d6b5-375b-416e-af3e-ecb51160691f" (UID: "9463d6b5-375b-416e-af3e-ecb51160691f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:51.691376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.691352 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:51.693878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.693853 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9463d6b5-375b-416e-af3e-ecb51160691f-kube-api-access-fz4gd" pod "9463d6b5-375b-416e-af3e-ecb51160691f" (UID: "9463d6b5-375b-416e-af3e-ecb51160691f"). InnerVolumeSpecName "kube-api-access-fz4gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:51.693983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.693966 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-xks-ca" pod "9463d6b5-375b-416e-af3e-ecb51160691f" (UID: "9463d6b5-375b-416e-af3e-ecb51160691f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:35:51.792814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.792777 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9463d6b5-375b-416e-af3e-ecb51160691f-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:51.792814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.792812 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4gd\" (UniqueName: \"kubernetes.io/projected/9463d6b5-375b-416e-af3e-ecb51160691f-kube-api-access-fz4gd\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:51.954400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.954298 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj"] Oct 30 13:35:51.955572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.955535 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9463d6b5-375b-416e-af3e-ecb51160691f" containerName="revoke-keys" Oct 30 13:35:51.955572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.955561 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9463d6b5-375b-416e-af3e-ecb51160691f" containerName="revoke-keys" Oct 30 13:35:51.955824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.955810 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9463d6b5-375b-416e-af3e-ecb51160691f" containerName="revoke-keys" Oct 30 13:35:51.958853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.958831 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 30 13:35:51.959194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.958872 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" podUID="88af2245-014a-4598-a442-7053d9d1e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 30 13:35:51.960354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.960337 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:51.976745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.976722 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 13:35:51.976824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.976763 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 13:35:51.976824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.976797 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-glws9\"" Oct 30 13:35:51.976824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.976802 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 13:35:51.991061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.991030 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj"] Oct 30 13:35:51.995772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.995743 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:51.995881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.995797 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:51.995940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.995914 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnpfr\" (UniqueName: \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-kube-api-access-pnpfr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:51.996068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.996045 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:51.996193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.996175 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:51.996297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.996211 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaee4df-ebf8-4573-828d-a9acbc068eed-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:51.996297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:51.996233 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.098469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.098430 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.098642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.098485 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.098642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.098541 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnpfr\" (UniqueName: \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-kube-api-access-pnpfr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.098642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.098610 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.098763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.098700 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.098763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.098741 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaee4df-ebf8-4573-828d-a9acbc068eed-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.098827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.098770 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.099263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.099236 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.101297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.101272 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.101297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.101279 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaee4df-ebf8-4573-828d-a9acbc068eed-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.101545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.101419 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.102546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.102507 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.136368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.136258 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.139102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.139072 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnpfr\" (UniqueName: \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-kube-api-access-pnpfr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv4kj\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.160683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.160661 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" Oct 30 13:35:52.160809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.160692 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s" event={"ID":"9463d6b5-375b-416e-af3e-ecb51160691f","Type":"ContainerDied","Data":"6e3c0ea7296f82f5fc714f07c90917e989242cf2ad4d3bcf78ff495a96b2eca5"} Oct 30 13:35:52.160809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.160743 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e3c0ea7296f82f5fc714f07c90917e989242cf2ad4d3bcf78ff495a96b2eca5" Oct 30 13:35:52.276873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.276786 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:52.613359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.605618 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:52.677701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.676242 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj"] Oct 30 13:35:52.752377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.752344 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:52.752483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.752408 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:52.964445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.963571 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 30 13:35:52.964445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:52.963649 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" podUID="88af2245-014a-4598-a442-7053d9d1e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 30 13:35:53.044963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.044556 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:35:53.159077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.158975 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-credential-socket\") pod \"a2e3f959-a61b-4e80-9437-4597939fe709\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " Oct 30 13:35:53.159243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.159108 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-envoy\") pod \"a2e3f959-a61b-4e80-9437-4597939fe709\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " Oct 30 13:35:53.159243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.159155 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e3f959-a61b-4e80-9437-4597939fe709-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e3f959-a61b-4e80-9437-4597939fe709-istiod-ca-cert\") pod \"a2e3f959-a61b-4e80-9437-4597939fe709\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " Oct 30 13:35:53.159385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.159252 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-kube-api-access-2gkm7\" (UniqueName: \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-kube-api-access-2gkm7\") pod \"a2e3f959-a61b-4e80-9437-4597939fe709\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " Oct 30 13:35:53.161978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.159480 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-socket\") pod \"a2e3f959-a61b-4e80-9437-4597939fe709\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " Oct 30 13:35:53.161978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.159563 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2e3f959-a61b-4e80-9437-4597939fe709-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e3f959-a61b-4e80-9437-4597939fe709-istio-podinfo\") pod \"a2e3f959-a61b-4e80-9437-4597939fe709\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " Oct 30 13:35:53.161978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.159566 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e3f959-a61b-4e80-9437-4597939fe709-istiod-ca-cert" pod "a2e3f959-a61b-4e80-9437-4597939fe709" (UID: "a2e3f959-a61b-4e80-9437-4597939fe709"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:53.161978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.159686 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-certs\") pod \"a2e3f959-a61b-4e80-9437-4597939fe709\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " Oct 30 13:35:53.161978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.159797 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-data\") pod \"a2e3f959-a61b-4e80-9437-4597939fe709\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " Oct 30 13:35:53.161978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.159838 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-istio-token\" (UniqueName: \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-istio-token\") pod \"a2e3f959-a61b-4e80-9437-4597939fe709\" (UID: \"a2e3f959-a61b-4e80-9437-4597939fe709\") " Oct 30 13:35:53.161978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.159857 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-credential-socket" pod "a2e3f959-a61b-4e80-9437-4597939fe709" (UID: "a2e3f959-a61b-4e80-9437-4597939fe709"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:53.161978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.161595 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.161978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.161627 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e3f959-a61b-4e80-9437-4597939fe709-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.162603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.162497 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-envoy" pod "a2e3f959-a61b-4e80-9437-4597939fe709" (UID: "a2e3f959-a61b-4e80-9437-4597939fe709"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:53.174663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.174621 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-socket" pod "a2e3f959-a61b-4e80-9437-4597939fe709" (UID: "a2e3f959-a61b-4e80-9437-4597939fe709"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:53.174808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.174714 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2e3f959-a61b-4e80-9437-4597939fe709-istio-podinfo" pod "a2e3f959-a61b-4e80-9437-4597939fe709" (UID: "a2e3f959-a61b-4e80-9437-4597939fe709"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:35:53.174886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.174841 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-kube-api-access-2gkm7" pod "a2e3f959-a61b-4e80-9437-4597939fe709" (UID: "a2e3f959-a61b-4e80-9437-4597939fe709"). InnerVolumeSpecName "kube-api-access-2gkm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:53.175097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.175067 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-certs" pod "a2e3f959-a61b-4e80-9437-4597939fe709" (UID: "a2e3f959-a61b-4e80-9437-4597939fe709"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:53.182939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.175261 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-data" pod "a2e3f959-a61b-4e80-9437-4597939fe709" (UID: "a2e3f959-a61b-4e80-9437-4597939fe709"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:53.190738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.190708 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-istio-token" pod "a2e3f959-a61b-4e80-9437-4597939fe709" (UID: "a2e3f959-a61b-4e80-9437-4597939fe709"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:53.199470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.199438 5049 generic.go:356] "Generic (PLEG): container finished" podID="a2e3f959-a61b-4e80-9437-4597939fe709" containerID="9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555" exitCode=0 Oct 30 13:35:53.199629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.199596 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" event={"ID":"a2e3f959-a61b-4e80-9437-4597939fe709","Type":"ContainerDied","Data":"9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555"} Oct 30 13:35:53.199708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.199697 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" event={"ID":"a2e3f959-a61b-4e80-9437-4597939fe709","Type":"ContainerDied","Data":"5998e384f6453244f9b8a4334188e5dab85018c12d3654b47dc50426186228fd"} Oct 30 13:35:53.199767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.199730 5049 scope.go:122] "RemoveContainer" containerID="9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555" Oct 30 13:35:53.199913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.199896 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf" Oct 30 13:35:53.249166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.249127 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" event={"ID":"9eaee4df-ebf8-4573-828d-a9acbc068eed","Type":"ContainerStarted","Data":"0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426"} Oct 30 13:35:53.249303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.249195 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" event={"ID":"9eaee4df-ebf8-4573-828d-a9acbc068eed","Type":"ContainerStarted","Data":"0816f934da39293b5c6728820d168f3c8e467a11b1098f738ccb031790bf5f83"} Oct 30 13:35:53.251668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.250731 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:53.254520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.254492 5049 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kv4kj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:35:53.254727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.254704 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" podUID="9eaee4df-ebf8-4573-828d-a9acbc068eed" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:35:53.257981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.257910 5049 scope.go:122] "RemoveContainer" containerID="27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830" Oct 30 13:35:53.263647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.263624 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.263766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.263754 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.263856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.263846 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.263931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.263923 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.264023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.264012 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gkm7\" (UniqueName: \"kubernetes.io/projected/a2e3f959-a61b-4e80-9437-4597939fe709-kube-api-access-2gkm7\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.264103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.264091 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e3f959-a61b-4e80-9437-4597939fe709-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.264189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.264178 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e3f959-a61b-4e80-9437-4597939fe709-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.271148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.271127 5049 scope.go:122] "RemoveContainer" containerID="9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555" Oct 30 13:35:53.271440 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:35:53.271417 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555\": container with ID starting with 9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555 not found: ID does not exist" containerID="9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555" Oct 30 13:35:53.271512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.271448 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555"} err="failed to get container status \"9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555\": rpc error: code = NotFound desc = could not find container \"9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555\": container with ID starting with 9a84c9e3007c24561dc260bd1c34361d7f4abd7f7a47dac962ef706d0dbd7555 not found: ID does not exist" Oct 30 13:35:53.271512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.271466 5049 scope.go:122] "RemoveContainer" containerID="27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830" Oct 30 13:35:53.271780 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:35:53.271754 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830\": container with ID starting with 27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830 not found: ID does not exist" containerID="27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830" Oct 30 13:35:53.271833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.271789 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830"} err="failed to get container status \"27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830\": rpc error: code = NotFound desc = could not find container \"27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830\": container with ID starting with 27fb8f6e11cf8eebae0481f30e23b3f93d24d554f39c518b0adb421425caa830 not found: ID does not exist" Oct 30 13:35:53.295685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.295637 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" podStartSLOduration=2.295623966 podStartE2EDuration="2.295623966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:53.292020483 +0000 UTC m=+2660.107214908" watchObservedRunningTime="2026-10-30 13:35:53.295623966 +0000 UTC m=+2660.110818389" Oct 30 13:35:53.357146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.357110 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf"] Oct 30 13:35:53.357146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.357150 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-cfsmf"] Oct 30 13:35:53.667641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.667597 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:53.829399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.828923 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:53.960341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.959754 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 30 13:35:53.960341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:53.959814 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" podUID="88af2245-014a-4598-a442-7053d9d1e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 30 13:35:54.149109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:54.148151 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r"] Oct 30 13:35:54.149109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:54.148583 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" containerID="cri-o://b80fc5b9991372b3f580d0b84c44a1c3cbecdad18ef9a0d921ffb46570486d1b" gracePeriod=30 Oct 30 13:35:54.277595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:54.270771 5049 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kv4kj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:35:54.277595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:54.270826 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" podUID="9eaee4df-ebf8-4573-828d-a9acbc068eed" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:35:54.763584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:54.763547 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:54.958737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:54.958704 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Oct 30 13:35:54.959001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:54.958980 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" podUID="88af2245-014a-4598-a442-7053d9d1e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Oct 30 13:35:55.280817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:55.280786 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:35:55.356710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:55.356676 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" path="/var/lib/kubelet/pods/a2e3f959-a61b-4e80-9437-4597939fe709/volumes" Oct 30 13:35:55.965298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:55.965269 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:55.965816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:55.965789 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:55.966640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:55.966622 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:35:56.636070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:56.636029 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:56.685753 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:35:56.685709 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:35:58.426348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:58.417954 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:58.525293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:58.525253 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h"] Oct 30 13:35:58.715232 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:58.715155 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:58.896602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:58.896562 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:59.373124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.372693 5049 generic.go:356] "Generic (PLEG): container finished" podID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerID="b80fc5b9991372b3f580d0b84c44a1c3cbecdad18ef9a0d921ffb46570486d1b" exitCode=0 Oct 30 13:35:59.373124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.373058 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" event={"ID":"48bc7804-9970-4bcf-90ed-fb7ef98b4471","Type":"ContainerDied","Data":"b80fc5b9991372b3f580d0b84c44a1c3cbecdad18ef9a0d921ffb46570486d1b"} Oct 30 13:35:59.570951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.570920 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:35:59.705388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.705261 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-kube-api-access-jqjm5\" (UniqueName: \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-kube-api-access-jqjm5\") pod \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " Oct 30 13:35:59.705388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.705315 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istiod-ca-cert\") pod \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " Oct 30 13:35:59.705654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.705400 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-token\" (UniqueName: \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-token\") pod \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " Oct 30 13:35:59.705654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.705427 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-data\") pod \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " Oct 30 13:35:59.705654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.705616 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-credential-socket\") pod \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " Oct 30 13:35:59.705896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.705657 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-envoy\") pod \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " Oct 30 13:35:59.705896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.705724 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-podinfo\") pod \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " Oct 30 13:35:59.705896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.705761 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-certs\") pod \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " Oct 30 13:35:59.705896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.705811 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-socket\") pod \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\" (UID: \"48bc7804-9970-4bcf-90ed-fb7ef98b4471\") " Oct 30 13:35:59.706444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.706410 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istiod-ca-cert" pod "48bc7804-9970-4bcf-90ed-fb7ef98b4471" (UID: "48bc7804-9970-4bcf-90ed-fb7ef98b4471"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:59.706713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.706687 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:59.706935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.706914 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-socket" pod "48bc7804-9970-4bcf-90ed-fb7ef98b4471" (UID: "48bc7804-9970-4bcf-90ed-fb7ef98b4471"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:59.707116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.707097 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-data" pod "48bc7804-9970-4bcf-90ed-fb7ef98b4471" (UID: "48bc7804-9970-4bcf-90ed-fb7ef98b4471"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:59.707201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.707133 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-credential-socket" pod "48bc7804-9970-4bcf-90ed-fb7ef98b4471" (UID: "48bc7804-9970-4bcf-90ed-fb7ef98b4471"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:59.707278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.707197 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-certs" pod "48bc7804-9970-4bcf-90ed-fb7ef98b4471" (UID: "48bc7804-9970-4bcf-90ed-fb7ef98b4471"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:59.709729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.709701 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-envoy" pod "48bc7804-9970-4bcf-90ed-fb7ef98b4471" (UID: "48bc7804-9970-4bcf-90ed-fb7ef98b4471"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:59.710117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.710095 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-kube-api-access-jqjm5" pod "48bc7804-9970-4bcf-90ed-fb7ef98b4471" (UID: "48bc7804-9970-4bcf-90ed-fb7ef98b4471"). InnerVolumeSpecName "kube-api-access-jqjm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:59.711507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.711482 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-podinfo" pod "48bc7804-9970-4bcf-90ed-fb7ef98b4471" (UID: "48bc7804-9970-4bcf-90ed-fb7ef98b4471"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:35:59.711766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.711744 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-token" pod "48bc7804-9970-4bcf-90ed-fb7ef98b4471" (UID: "48bc7804-9970-4bcf-90ed-fb7ef98b4471"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:59.808336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.808288 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:59.808336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.808336 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:59.808336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.808347 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:59.808662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.808355 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:59.808662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.808364 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:59.808662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.808372 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqjm5\" (UniqueName: \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-kube-api-access-jqjm5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:59.808662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.808382 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:35:59.808662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:35:59.808390 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bc7804-9970-4bcf-90ed-fb7ef98b4471-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:00.032885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:00.032779 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn"] Oct 30 13:36:00.037834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:00.037804 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366b8d90-8npfn"] Oct 30 13:36:00.383954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:00.383918 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" event={"ID":"48bc7804-9970-4bcf-90ed-fb7ef98b4471","Type":"ContainerDied","Data":"1d40665a5680c5c93f640107705af89aedc6b0f0f0237235e091d843ea01581e"} Oct 30 13:36:00.383954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:00.383942 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r" Oct 30 13:36:00.384227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:00.383987 5049 scope.go:122] "RemoveContainer" containerID="b80fc5b9991372b3f580d0b84c44a1c3cbecdad18ef9a0d921ffb46570486d1b" Oct 30 13:36:00.443216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:00.443180 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r"] Oct 30 13:36:00.453594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:00.453557 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-kmj6r"] Oct 30 13:36:00.477507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:00.477411 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:00.903974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:00.903941 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:01.342574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:01.342541 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35549aa7-a70b-419c-819b-aa5bdabc1036" path="/var/lib/kubelet/pods/35549aa7-a70b-419c-819b-aa5bdabc1036/volumes" Oct 30 13:36:01.343199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:01.343181 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" path="/var/lib/kubelet/pods/48bc7804-9970-4bcf-90ed-fb7ef98b4471/volumes" Oct 30 13:36:01.587695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:01.587663 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:02.108580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:02.108541 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:02.467651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:02.467557 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:02.625266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:02.625222 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:03.526304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:03.526263 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" podUID="7c02acd4-e3a0-4105-bdd7-73089832058b" containerName="payload-processing" containerID="cri-o://fb48f962aacbd9f736c92f0feb005b0c1bb56462e23f750e6df2730c2efc682a" gracePeriod=25 Oct 30 13:36:04.329602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:04.329381 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:05.647244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:05.647210 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:05.804453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:05.804415 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:06.345221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.345182 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" podUID="17fd01c8-8e29-445d-be8f-156b8159f469" containerName="main" containerID="cri-o://40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1" gracePeriod=45 Oct 30 13:36:06.559123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.559090 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:06.706763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.706731 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:36:06.791663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.791627 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-model-cache\") pod \"17fd01c8-8e29-445d-be8f-156b8159f469\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " Oct 30 13:36:06.791872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.791704 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-home\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-home\") pod \"17fd01c8-8e29-445d-be8f-156b8159f469\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " Oct 30 13:36:06.791872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.791804 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-tmp-dir\") pod \"17fd01c8-8e29-445d-be8f-156b8159f469\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " Oct 30 13:36:06.791998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.791891 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-model-cache" pod "17fd01c8-8e29-445d-be8f-156b8159f469" (UID: "17fd01c8-8e29-445d-be8f-156b8159f469"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:06.792061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.791992 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17fd01c8-8e29-445d-be8f-156b8159f469-tls-certs\" (UniqueName: \"kubernetes.io/secret/17fd01c8-8e29-445d-be8f-156b8159f469-tls-certs\") pod \"17fd01c8-8e29-445d-be8f-156b8159f469\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " Oct 30 13:36:06.792061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.791996 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-home" pod "17fd01c8-8e29-445d-be8f-156b8159f469" (UID: "17fd01c8-8e29-445d-be8f-156b8159f469"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:06.792061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.792037 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-dshm\") pod \"17fd01c8-8e29-445d-be8f-156b8159f469\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " Oct 30 13:36:06.792229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.792106 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fd01c8-8e29-445d-be8f-156b8159f469-kube-api-access-4qm87\" (UniqueName: \"kubernetes.io/projected/17fd01c8-8e29-445d-be8f-156b8159f469-kube-api-access-4qm87\") pod \"17fd01c8-8e29-445d-be8f-156b8159f469\" (UID: \"17fd01c8-8e29-445d-be8f-156b8159f469\") " Oct 30 13:36:06.792229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.792133 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-tmp-dir" pod "17fd01c8-8e29-445d-be8f-156b8159f469" (UID: "17fd01c8-8e29-445d-be8f-156b8159f469"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:06.792938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.792911 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:06.793223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.793207 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:06.793367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.793352 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:06.794681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.794657 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fd01c8-8e29-445d-be8f-156b8159f469-tls-certs" pod "17fd01c8-8e29-445d-be8f-156b8159f469" (UID: "17fd01c8-8e29-445d-be8f-156b8159f469"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:36:06.794976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.794884 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-dshm" pod "17fd01c8-8e29-445d-be8f-156b8159f469" (UID: "17fd01c8-8e29-445d-be8f-156b8159f469"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:06.795097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.794926 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fd01c8-8e29-445d-be8f-156b8159f469-kube-api-access-4qm87" pod "17fd01c8-8e29-445d-be8f-156b8159f469" (UID: "17fd01c8-8e29-445d-be8f-156b8159f469"). InnerVolumeSpecName "kube-api-access-4qm87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:06.895103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.895071 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17fd01c8-8e29-445d-be8f-156b8159f469-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:06.895103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.895101 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17fd01c8-8e29-445d-be8f-156b8159f469-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:06.895103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:06.895110 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qm87\" (UniqueName: \"kubernetes.io/projected/17fd01c8-8e29-445d-be8f-156b8159f469-kube-api-access-4qm87\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:07.299563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.299525 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:07.461960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.461926 5049 generic.go:356] "Generic (PLEG): container finished" podID="17fd01c8-8e29-445d-be8f-156b8159f469" containerID="40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1" exitCode=0 Oct 30 13:36:07.462177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.461970 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" Oct 30 13:36:07.462177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.461980 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" event={"ID":"17fd01c8-8e29-445d-be8f-156b8159f469","Type":"ContainerDied","Data":"40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1"} Oct 30 13:36:07.462177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.462017 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr" event={"ID":"17fd01c8-8e29-445d-be8f-156b8159f469","Type":"ContainerDied","Data":"62104196cb37b2ad116b152ee7444957fe4eefe2482b8f0c9cf02a196387848f"} Oct 30 13:36:07.462177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.462035 5049 scope.go:122] "RemoveContainer" containerID="40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1" Oct 30 13:36:07.464683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.464150 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:07.479259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.479236 5049 scope.go:122] "RemoveContainer" containerID="40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1" Oct 30 13:36:07.479657 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:36:07.479635 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1\": container with ID starting with 40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1 not found: ID does not exist" containerID="40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1" Oct 30 13:36:07.479765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.479663 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1"} err="failed to get container status \"40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1\": rpc error: code = NotFound desc = could not find container \"40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1\": container with ID starting with 40cd987c5271b3fa2b8a95beea7c40bce713418fafa97eb97754ea953a6babe1 not found: ID does not exist" Oct 30 13:36:07.606395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.605304 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr"] Oct 30 13:36:07.752162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:07.752048 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9cbcbc/auth-test-model-9cbcbc-kserve-5cb86b95c7-267tr"] Oct 30 13:36:08.387297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:08.381133 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:08.813219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:08.813168 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7"] Oct 30 13:36:08.881845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:08.881818 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:09.348653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:09.348616 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17fd01c8-8e29-445d-be8f-156b8159f469" path="/var/lib/kubelet/pods/17fd01c8-8e29-445d-be8f-156b8159f469/volumes" Oct 30 13:36:10.397574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:10.397546 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:10.551999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:10.551966 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:11.033947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:11.033910 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c"] Oct 30 13:36:11.040834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:11.040803 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb64431-k2f2c"] Oct 30 13:36:11.342963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:11.342931 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb21d7cf-6717-4154-911b-86802f32fd4c" path="/var/lib/kubelet/pods/eb21d7cf-6717-4154-911b-86802f32fd4c/volumes" Oct 30 13:36:11.496450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:11.496411 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" podUID="7c02acd4-e3a0-4105-bdd7-73089832058b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:36:12.605012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:12.604976 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:14.573626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:14.573596 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:14.852403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:14.851526 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:16.816481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:16.816433 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:36:16.917754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:16.917716 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:36:17.329985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:17.329949 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:18.039955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:18.039921 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk"] Oct 30 13:36:18.045307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:18.045278 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c746f1a-rw8wk"] Oct 30 13:36:19.132045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:19.132013 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:19.344831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:19.344799 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754e84ac-3785-4b17-962c-f555525caf40" path="/var/lib/kubelet/pods/754e84ac-3785-4b17-962c-f555525caf40/volumes" Oct 30 13:36:20.876946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:20.876910 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:21.272435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:21.272302 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:21.497239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:21.497195 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" podUID="7c02acd4-e3a0-4105-bdd7-73089832058b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:36:21.878201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:21.878164 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:22.342110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:22.342076 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:22.530994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:22.530914 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:22.937342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:22.936856 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:23.843055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:23.842514 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" podUID="68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" containerName="main" containerID="cri-o://24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2" gracePeriod=45 Oct 30 13:36:23.858810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:23.858651 5049 generic.go:356] "Generic (PLEG): container finished" podID="7c02acd4-e3a0-4105-bdd7-73089832058b" containerID="fb48f962aacbd9f736c92f0feb005b0c1bb56462e23f750e6df2730c2efc682a" exitCode=0 Oct 30 13:36:23.858810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:23.858735 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" event={"ID":"7c02acd4-e3a0-4105-bdd7-73089832058b","Type":"ContainerDied","Data":"fb48f962aacbd9f736c92f0feb005b0c1bb56462e23f750e6df2730c2efc682a"} Oct 30 13:36:24.014547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.014514 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:36:24.075471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.075436 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c02acd4-e3a0-4105-bdd7-73089832058b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c02acd4-e3a0-4105-bdd7-73089832058b-plugins-config-volume\") pod \"7c02acd4-e3a0-4105-bdd7-73089832058b\" (UID: \"7c02acd4-e3a0-4105-bdd7-73089832058b\") " Oct 30 13:36:24.075861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.075839 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c02acd4-e3a0-4105-bdd7-73089832058b-kube-api-access-h4w48\" (UniqueName: \"kubernetes.io/projected/7c02acd4-e3a0-4105-bdd7-73089832058b-kube-api-access-h4w48\") pod \"7c02acd4-e3a0-4105-bdd7-73089832058b\" (UID: \"7c02acd4-e3a0-4105-bdd7-73089832058b\") " Oct 30 13:36:24.077059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.077034 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c02acd4-e3a0-4105-bdd7-73089832058b-plugins-config-volume" pod "7c02acd4-e3a0-4105-bdd7-73089832058b" (UID: "7c02acd4-e3a0-4105-bdd7-73089832058b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:36:24.085408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.080904 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c02acd4-e3a0-4105-bdd7-73089832058b-kube-api-access-h4w48" pod "7c02acd4-e3a0-4105-bdd7-73089832058b" (UID: "7c02acd4-e3a0-4105-bdd7-73089832058b"). InnerVolumeSpecName "kube-api-access-h4w48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:24.178382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.178297 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4w48\" (UniqueName: \"kubernetes.io/projected/7c02acd4-e3a0-4105-bdd7-73089832058b-kube-api-access-h4w48\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:24.178382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.178355 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c02acd4-e3a0-4105-bdd7-73089832058b-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:24.220494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.220461 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:36:24.279499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.279464 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-kube-api-access-nwdjl\" (UniqueName: \"kubernetes.io/projected/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-kube-api-access-nwdjl\") pod \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " Oct 30 13:36:24.279755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.279682 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tmp-dir\") pod \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " Oct 30 13:36:24.279755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.279719 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-home\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-home\") pod \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " Oct 30 13:36:24.279887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.279837 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tls-certs\" (UniqueName: \"kubernetes.io/secret/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tls-certs\") pod \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " Oct 30 13:36:24.279887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.279875 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-dshm\") pod \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " Oct 30 13:36:24.279989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.279925 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-model-cache\") pod \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\" (UID: \"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f\") " Oct 30 13:36:24.280251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.280228 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-home" pod "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" (UID: "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:24.280657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.280612 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:24.281075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.281049 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-model-cache" pod "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" (UID: "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:24.281445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.281421 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tmp-dir" pod "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" (UID: "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:24.284453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.284431 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-kube-api-access-nwdjl" pod "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" (UID: "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f"). InnerVolumeSpecName "kube-api-access-nwdjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:24.285063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.285035 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-dshm" pod "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" (UID: "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:36:24.285169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.285106 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tls-certs" pod "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" (UID: "68fc0e80-aed3-4ebc-8a2e-78ffba42e75f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:36:24.382455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.382346 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:24.382455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.382381 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:24.382455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.382397 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:24.382455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.382409 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:24.382455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.382420 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwdjl\" (UniqueName: \"kubernetes.io/projected/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f-kube-api-access-nwdjl\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:24.870371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.870340 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" Oct 30 13:36:24.870608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.870375 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h" event={"ID":"7c02acd4-e3a0-4105-bdd7-73089832058b","Type":"ContainerDied","Data":"ba4a1f3ff087acbbcd6fe0cf88b21859e908d1df10e44bd0d79089fde64a3d0e"} Oct 30 13:36:24.870608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.870437 5049 scope.go:122] "RemoveContainer" containerID="fb48f962aacbd9f736c92f0feb005b0c1bb56462e23f750e6df2730c2efc682a" Oct 30 13:36:24.872174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.872149 5049 generic.go:356] "Generic (PLEG): container finished" podID="68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" containerID="24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2" exitCode=0 Oct 30 13:36:24.872286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.872194 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" Oct 30 13:36:24.872286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.872221 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" event={"ID":"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f","Type":"ContainerDied","Data":"24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2"} Oct 30 13:36:24.872286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.872264 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7" event={"ID":"68fc0e80-aed3-4ebc-8a2e-78ffba42e75f","Type":"ContainerDied","Data":"0e6ac932db1f02c6a8eae618f6ba9377232701d3eb98e3d124ca50e878b4d85b"} Oct 30 13:36:24.883678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.883658 5049 scope.go:122] "RemoveContainer" containerID="24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2" Oct 30 13:36:24.898995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.898965 5049 scope.go:122] "RemoveContainer" containerID="24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2" Oct 30 13:36:24.899534 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:36:24.899502 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2\": container with ID starting with 24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2 not found: ID does not exist" containerID="24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2" Oct 30 13:36:24.899719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.899543 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2"} err="failed to get container status \"24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2\": rpc error: code = NotFound desc = could not find container \"24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2\": container with ID starting with 24c8691b8b3ae7197cf4312a42b7bf56171be6fcd1aa981790220d1d6a694bf2 not found: ID does not exist" Oct 30 13:36:24.902087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.902042 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:24.925015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.923653 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h"] Oct 30 13:36:24.929301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.929267 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b29df4-c97d768f6-sb52h"] Oct 30 13:36:24.964495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.964457 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7"] Oct 30 13:36:24.970359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:24.970304 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/rate-test-model-ff4285-kserve-5bc466f6cc-xmfw7"] Oct 30 13:36:25.340188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:25.340156 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" path="/var/lib/kubelet/pods/68fc0e80-aed3-4ebc-8a2e-78ffba42e75f/volumes" Oct 30 13:36:25.340692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:25.340676 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c02acd4-e3a0-4105-bdd7-73089832058b" path="/var/lib/kubelet/pods/7c02acd4-e3a0-4105-bdd7-73089832058b/volumes" Oct 30 13:36:26.079554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:26.079511 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:26.274866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:26.274835 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:26.831148 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:36:26.831089 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:36:26.919600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:26.919541 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:27.701170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:27.700936 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": read tcp 10.132.0.2:50028->10.132.1.31:15021: read: connection reset by peer" start-of-body= Oct 30 13:36:27.701390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:27.701210 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": read tcp 10.132.0.2:50028->10.132.1.31:15021: read: connection reset by peer" Oct 30 13:36:27.904693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:27.904661 5049 generic.go:356] "Generic (PLEG): container finished" podID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerID="2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5" exitCode=0 Oct 30 13:36:27.905191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:27.904735 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" event={"ID":"c491c8a2-4e07-4851-aa5d-6a55dc247c59","Type":"ContainerDied","Data":"2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5"} Oct 30 13:36:27.905286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:27.905264 5049 scope.go:122] "RemoveContainer" containerID="2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5" Oct 30 13:36:27.907828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:27.907798 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:27.907913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:27.907885 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:27.907955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:27.907932 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:28.034231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:28.034200 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk"] Oct 30 13:36:28.040655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:28.040622 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33efb373-j9whk"] Oct 30 13:36:28.564559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:28.564529 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:28.766461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:28.766401 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:28.964300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:28.960729 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" event={"ID":"c491c8a2-4e07-4851-aa5d-6a55dc247c59","Type":"ContainerStarted","Data":"127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf"} Oct 30 13:36:29.057065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:29.057037 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:29.353484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:29.353447 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273bd15c-ba75-4eca-8fa6-3dfea8324ee3" path="/var/lib/kubelet/pods/273bd15c-ba75-4eca-8fa6-3dfea8324ee3/volumes" Oct 30 13:36:29.405807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:29.405772 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:29.515342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:29.513710 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:36:29.515342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:29.513834 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 13:36:29.515342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:29.513879 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 13:36:30.524347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:30.509721 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 13:36:30.524347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:30.509787 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 13:36:30.907648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:30.905761 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:31.052382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:31.052338 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5"] Oct 30 13:36:31.261423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:31.239148 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:31.390259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:31.390224 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:31.516402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:31.506226 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 13:36:31.516402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:31.506296 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 13:36:31.979381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:31.979243 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:32.086282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:32.086247 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs"] Oct 30 13:36:32.255701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:32.255583 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:32.489731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:32.489695 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:32.541109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:32.541019 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 13:36:32.541494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:32.541452 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 13:36:33.035377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:33.032111 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:33.523020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:33.522570 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 13:36:33.523020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:33.522634 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 13:36:33.926345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:33.926271 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:36:33.951768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:33.951739 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:36:34.513791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:34.513748 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 13:36:34.514441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:34.514408 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 13:36:35.083157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:35.083115 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:35.510534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:35.510441 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:36:35.510918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:35.510894 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:36:35.511761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:35.511736 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:36:36.053103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.053064 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" podUID="73e43b89-8a7c-4a7b-9094-51364aa7bd85" containerName="payload-pre-processing" containerID="cri-o://eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30" gracePeriod=25 Oct 30 13:36:36.363394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.363362 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:36:36.407549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.407513 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e43b89-8a7c-4a7b-9094-51364aa7bd85-kube-api-access-wwcjk\" (UniqueName: \"kubernetes.io/projected/73e43b89-8a7c-4a7b-9094-51364aa7bd85-kube-api-access-wwcjk\") pod \"73e43b89-8a7c-4a7b-9094-51364aa7bd85\" (UID: \"73e43b89-8a7c-4a7b-9094-51364aa7bd85\") " Oct 30 13:36:36.408039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.408016 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e43b89-8a7c-4a7b-9094-51364aa7bd85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e43b89-8a7c-4a7b-9094-51364aa7bd85-plugins-config-volume\") pod \"73e43b89-8a7c-4a7b-9094-51364aa7bd85\" (UID: \"73e43b89-8a7c-4a7b-9094-51364aa7bd85\") " Oct 30 13:36:36.409055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.409026 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e43b89-8a7c-4a7b-9094-51364aa7bd85-plugins-config-volume" pod "73e43b89-8a7c-4a7b-9094-51364aa7bd85" (UID: "73e43b89-8a7c-4a7b-9094-51364aa7bd85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:36:36.411804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.411770 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e43b89-8a7c-4a7b-9094-51364aa7bd85-kube-api-access-wwcjk" pod "73e43b89-8a7c-4a7b-9094-51364aa7bd85" (UID: "73e43b89-8a7c-4a7b-9094-51364aa7bd85"). InnerVolumeSpecName "kube-api-access-wwcjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:36.510443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.510407 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e43b89-8a7c-4a7b-9094-51364aa7bd85-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.510443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.510438 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwcjk\" (UniqueName: \"kubernetes.io/projected/73e43b89-8a7c-4a7b-9094-51364aa7bd85-kube-api-access-wwcjk\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:36.674175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.674087 5049 scope.go:122] "RemoveContainer" containerID="429d936f990e6bfde7480e58927b7ae5fd690af40c85382a796276bd9deb6460" Oct 30 13:36:36.686215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.686191 5049 scope.go:122] "RemoveContainer" containerID="b0d9326272248f3ec62cbf9ef886e85656120d4c0d6636b0e1d519a34e2874c8" Oct 30 13:36:36.697086 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.697061 5049 scope.go:122] "RemoveContainer" containerID="5ae47b44ca7e275fcc894da5bef348d455c2cc968ccde642d550471f65815bde" Oct 30 13:36:36.709556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.709532 5049 scope.go:122] "RemoveContainer" containerID="2841ee77a5ac8f0e9754184867efd7d3fec032f9748362cecc48df829b575806" Oct 30 13:36:36.912512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:36.910546 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:37.087592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.087552 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" podUID="0031597f-732c-4d8c-bada-e6c47ce81c42" containerName="payload-pre-processing" containerID="cri-o://46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757" gracePeriod=25 Oct 30 13:36:37.110638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.110604 5049 generic.go:356] "Generic (PLEG): container finished" podID="73e43b89-8a7c-4a7b-9094-51364aa7bd85" containerID="eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30" exitCode=0 Oct 30 13:36:37.110805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.110663 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" Oct 30 13:36:37.111255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.111226 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" event={"ID":"73e43b89-8a7c-4a7b-9094-51364aa7bd85","Type":"ContainerDied","Data":"eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30"} Oct 30 13:36:37.111391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.111279 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5" event={"ID":"73e43b89-8a7c-4a7b-9094-51364aa7bd85","Type":"ContainerDied","Data":"2e9644a12c6c7b24c1d4fb0090508745f2aa59902ec65864eaaf42d76929aa67"} Oct 30 13:36:37.111391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.111375 5049 scope.go:122] "RemoveContainer" containerID="eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30" Oct 30 13:36:37.202415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.202395 5049 scope.go:122] "RemoveContainer" containerID="eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30" Oct 30 13:36:37.202686 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:36:37.202664 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30\": container with ID starting with eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30 not found: ID does not exist" containerID="eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30" Oct 30 13:36:37.202773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.202693 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30"} err="failed to get container status \"eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30\": rpc error: code = NotFound desc = could not find container \"eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30\": container with ID starting with eb8b6b72f3b4f37a5ae4f8d257e32d121eba5acf42366d108feb2de9bb9f3e30 not found: ID does not exist" Oct 30 13:36:37.232881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.232846 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5"] Oct 30 13:36:37.244795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.244758 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-8db79f95-mhcl5"] Oct 30 13:36:37.349310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.349227 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e43b89-8a7c-4a7b-9094-51364aa7bd85" path="/var/lib/kubelet/pods/73e43b89-8a7c-4a7b-9094-51364aa7bd85/volumes" Oct 30 13:36:37.414757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.414729 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:36:37.529578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.529544 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0031597f-732c-4d8c-bada-e6c47ce81c42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0031597f-732c-4d8c-bada-e6c47ce81c42-plugins-config-volume\") pod \"0031597f-732c-4d8c-bada-e6c47ce81c42\" (UID: \"0031597f-732c-4d8c-bada-e6c47ce81c42\") " Oct 30 13:36:37.529786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.529621 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0031597f-732c-4d8c-bada-e6c47ce81c42-kube-api-access-fc7tn\" (UniqueName: \"kubernetes.io/projected/0031597f-732c-4d8c-bada-e6c47ce81c42-kube-api-access-fc7tn\") pod \"0031597f-732c-4d8c-bada-e6c47ce81c42\" (UID: \"0031597f-732c-4d8c-bada-e6c47ce81c42\") " Oct 30 13:36:37.529964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.529937 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0031597f-732c-4d8c-bada-e6c47ce81c42-plugins-config-volume" pod "0031597f-732c-4d8c-bada-e6c47ce81c42" (UID: "0031597f-732c-4d8c-bada-e6c47ce81c42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:36:37.530469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.530444 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0031597f-732c-4d8c-bada-e6c47ce81c42-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:37.531903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.531874 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0031597f-732c-4d8c-bada-e6c47ce81c42-kube-api-access-fc7tn" pod "0031597f-732c-4d8c-bada-e6c47ce81c42" (UID: "0031597f-732c-4d8c-bada-e6c47ce81c42"). InnerVolumeSpecName "kube-api-access-fc7tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:37.632492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:37.632395 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc7tn\" (UniqueName: \"kubernetes.io/projected/0031597f-732c-4d8c-bada-e6c47ce81c42-kube-api-access-fc7tn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:38.137945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:38.137905 5049 generic.go:356] "Generic (PLEG): container finished" podID="0031597f-732c-4d8c-bada-e6c47ce81c42" containerID="46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757" exitCode=0 Oct 30 13:36:38.138476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:38.137974 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" Oct 30 13:36:38.138476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:38.138000 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" event={"ID":"0031597f-732c-4d8c-bada-e6c47ce81c42","Type":"ContainerDied","Data":"46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757"} Oct 30 13:36:38.138476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:38.138096 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs" event={"ID":"0031597f-732c-4d8c-bada-e6c47ce81c42","Type":"ContainerDied","Data":"af4c55f6335c24307d1379a7aa84cdb4b3c548b6a4b8715ea628a0bb0690522c"} Oct 30 13:36:38.138476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:38.138111 5049 scope.go:122] "RemoveContainer" containerID="46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757" Oct 30 13:36:38.153423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:38.153401 5049 scope.go:122] "RemoveContainer" containerID="46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757" Oct 30 13:36:38.153743 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:36:38.153721 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757\": container with ID starting with 46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757 not found: ID does not exist" containerID="46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757" Oct 30 13:36:38.153882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:38.153751 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757"} err="failed to get container status \"46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757\": rpc error: code = NotFound desc = could not find container \"46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757\": container with ID starting with 46cca3874b60bdb4df923cee64eef5a6b0b03a28d28ea38a0e80110bd71cd757 not found: ID does not exist" Oct 30 13:36:38.187759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:38.187731 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs"] Oct 30 13:36:38.196866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:38.196837 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-78bbc9cc5-cbmgs"] Oct 30 13:36:38.870672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:38.870626 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:39.034143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:39.034098 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx"] Oct 30 13:36:39.040755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:39.040713 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f443e7c-5xbkx"] Oct 30 13:36:39.343124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:39.343091 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0031597f-732c-4d8c-bada-e6c47ce81c42" path="/var/lib/kubelet/pods/0031597f-732c-4d8c-bada-e6c47ce81c42/volumes" Oct 30 13:36:39.343814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:39.343792 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f1b9ca-dcd0-411b-b2c3-cee56accd687" path="/var/lib/kubelet/pods/d9f1b9ca-dcd0-411b-b2c3-cee56accd687/volumes" Oct 30 13:36:41.173842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:41.173807 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:43.755921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:43.755876 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:45.357411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:45.357295 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:46.075346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:46.068724 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:46.184646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:46.184616 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s"] Oct 30 13:36:46.197445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:46.197412 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b29df4-zn94s"] Oct 30 13:36:46.231365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:46.230814 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:46.741868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:46.741831 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:47.353881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:47.353837 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9463d6b5-375b-416e-af3e-ecb51160691f" path="/var/lib/kubelet/pods/9463d6b5-375b-416e-af3e-ecb51160691f/volumes" Oct 30 13:36:48.532043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:48.531967 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:49.053696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:49.053663 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn"] Oct 30 13:36:49.060276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:49.060245 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6ce356-24gfn"] Oct 30 13:36:49.342312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:49.342227 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd59d610-f816-43e2-8d11-59286b028201" path="/var/lib/kubelet/pods/dd59d610-f816-43e2-8d11-59286b028201/volumes" Oct 30 13:36:49.956473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:49.956435 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:36:49.957008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:49.956499 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:36:50.951782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:50.951750 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:50.969738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:50.968472 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:36:50.969738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:50.968569 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:36:52.723191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:52.718910 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:53.488761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:53.488724 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:53.659668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:53.659635 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:53.886682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:53.886631 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:54.356332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.356279 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:54.499733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.499695 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk"] Oct 30 13:36:54.500727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500701 5049 cpu_manager.go:414] "containerMap: removing container" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" Oct 30 13:36:54.500727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500727 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500744 5049 cpu_manager.go:414] "containerMap: removing container" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500751 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500784 5049 cpu_manager.go:414] "containerMap: removing container" podUID="73e43b89-8a7c-4a7b-9094-51364aa7bd85" containerName="payload-pre-processing" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500792 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e43b89-8a7c-4a7b-9094-51364aa7bd85" containerName="payload-pre-processing" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500823 5049 cpu_manager.go:414] "containerMap: removing container" podUID="0031597f-732c-4d8c-bada-e6c47ce81c42" containerName="payload-pre-processing" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500830 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0031597f-732c-4d8c-bada-e6c47ce81c42" containerName="payload-pre-processing" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500856 5049 cpu_manager.go:414] "containerMap: removing container" podUID="68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" containerName="main" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500863 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" containerName="main" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500876 5049 cpu_manager.go:414] "containerMap: removing container" podUID="17fd01c8-8e29-445d-be8f-156b8159f469" containerName="main" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500883 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fd01c8-8e29-445d-be8f-156b8159f469" containerName="main" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500903 5049 cpu_manager.go:414] "containerMap: removing container" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500910 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500929 5049 cpu_manager.go:414] "containerMap: removing container" podUID="7c02acd4-e3a0-4105-bdd7-73089832058b" containerName="payload-processing" Oct 30 13:36:54.500939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.500936 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c02acd4-e3a0-4105-bdd7-73089832058b" containerName="payload-processing" Oct 30 13:36:54.501693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.501159 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" Oct 30 13:36:54.501693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.501178 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e3f959-a61b-4e80-9437-4597939fe709" containerName="istio-proxy" Oct 30 13:36:54.501693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.501191 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e43b89-8a7c-4a7b-9094-51364aa7bd85" containerName="payload-pre-processing" Oct 30 13:36:54.501693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.501206 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="0031597f-732c-4d8c-bada-e6c47ce81c42" containerName="payload-pre-processing" Oct 30 13:36:54.501693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.501221 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fc0e80-aed3-4ebc-8a2e-78ffba42e75f" containerName="main" Oct 30 13:36:54.501693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.501232 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fd01c8-8e29-445d-be8f-156b8159f469" containerName="main" Oct 30 13:36:54.501693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.501244 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bc7804-9970-4bcf-90ed-fb7ef98b4471" containerName="istio-proxy" Oct 30 13:36:54.501693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.501254 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c02acd4-e3a0-4105-bdd7-73089832058b" containerName="payload-processing" Oct 30 13:36:54.618877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.581269 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk"] Oct 30 13:36:54.618877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.593647 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:54.618877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.603031 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:54.618877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.614464 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:36:54.618877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.616092 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:36:54.803199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.803162 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:54.803414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.803223 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6g7f\" (UniqueName: \"kubernetes.io/projected/36df70c3-5c45-4573-9c0e-998eaa64c614-kube-api-access-f6g7f\") pod \"maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:54.803494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.803432 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:54.904597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.904525 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:54.905029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.904710 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:54.905029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.904749 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6g7f\" (UniqueName: \"kubernetes.io/projected/36df70c3-5c45-4573-9c0e-998eaa64c614-kube-api-access-f6g7f\") pod \"maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:54.905537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.905506 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:54.907956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.907922 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:54.927450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.927419 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6g7f\" (UniqueName: \"kubernetes.io/projected/36df70c3-5c45-4573-9c0e-998eaa64c614-kube-api-access-f6g7f\") pod \"maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:54.954910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:54.954872 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:55.615982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:55.615949 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk"] Oct 30 13:36:55.618525 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:36:55.618496 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36df70c3_5c45_4573_9c0e_998eaa64c614.slice/crio-d861385065f00a37faea55d24ef5a835fa4da53e04af8eff9d805a5a82193494 WatchSource:0}: Error finding container d861385065f00a37faea55d24ef5a835fa4da53e04af8eff9d805a5a82193494: Status 404 returned error can't find the container with id d861385065f00a37faea55d24ef5a835fa4da53e04af8eff9d805a5a82193494 Oct 30 13:36:56.529794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:56.529702 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" event={"ID":"36df70c3-5c45-4573-9c0e-998eaa64c614","Type":"ContainerStarted","Data":"f015dc2b01d074d1c9873d2c78017744f1b42715c9f5833c7738c3f1a02aa9e4"} Oct 30 13:36:56.529794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:56.529772 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" event={"ID":"36df70c3-5c45-4573-9c0e-998eaa64c614","Type":"ContainerStarted","Data":"d861385065f00a37faea55d24ef5a835fa4da53e04af8eff9d805a5a82193494"} Oct 30 13:36:56.573625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:56.573565 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" podStartSLOduration=2.573543339 podStartE2EDuration="2.573543339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:56.568020372 +0000 UTC m=+2723.383214798" watchObservedRunningTime="2026-10-30 13:36:56.573543339 +0000 UTC m=+2723.388737767" Oct 30 13:36:56.593012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:56.592975 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:56.756225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:56.756164 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:56.914514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:56.914434 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:56.972682 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:36:56.972635 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:36:57.545392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:57.545310 5049 generic.go:356] "Generic (PLEG): container finished" podID="36df70c3-5c45-4573-9c0e-998eaa64c614" containerID="f015dc2b01d074d1c9873d2c78017744f1b42715c9f5833c7738c3f1a02aa9e4" exitCode=0 Oct 30 13:36:57.546138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:57.546111 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" event={"ID":"36df70c3-5c45-4573-9c0e-998eaa64c614","Type":"ContainerDied","Data":"f015dc2b01d074d1c9873d2c78017744f1b42715c9f5833c7738c3f1a02aa9e4"} Oct 30 13:36:57.734561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:57.734521 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:57.914762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:57.914587 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:58.745479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:58.740186 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:58.866742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:58.866710 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:58.924340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:58.919862 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-openshift-ca\") pod \"36df70c3-5c45-4573-9c0e-998eaa64c614\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " Oct 30 13:36:58.924340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:58.920273 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-openshift-ca" pod "36df70c3-5c45-4573-9c0e-998eaa64c614" (UID: "36df70c3-5c45-4573-9c0e-998eaa64c614"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:36:58.924340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:58.920459 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36df70c3-5c45-4573-9c0e-998eaa64c614-kube-api-access-f6g7f\" (UniqueName: \"kubernetes.io/projected/36df70c3-5c45-4573-9c0e-998eaa64c614-kube-api-access-f6g7f\") pod \"36df70c3-5c45-4573-9c0e-998eaa64c614\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " Oct 30 13:36:58.924340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:58.921066 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-xks-ca\") pod \"36df70c3-5c45-4573-9c0e-998eaa64c614\" (UID: \"36df70c3-5c45-4573-9c0e-998eaa64c614\") " Oct 30 13:36:58.924340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:58.922242 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:58.924340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:58.923927 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36df70c3-5c45-4573-9c0e-998eaa64c614-kube-api-access-f6g7f" pod "36df70c3-5c45-4573-9c0e-998eaa64c614" (UID: "36df70c3-5c45-4573-9c0e-998eaa64c614"). InnerVolumeSpecName "kube-api-access-f6g7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:58.937382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:58.936435 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-xks-ca" pod "36df70c3-5c45-4573-9c0e-998eaa64c614" (UID: "36df70c3-5c45-4573-9c0e-998eaa64c614"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:36:59.023949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:59.023912 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6g7f\" (UniqueName: \"kubernetes.io/projected/36df70c3-5c45-4573-9c0e-998eaa64c614-kube-api-access-f6g7f\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:59.023949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:59.023944 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36df70c3-5c45-4573-9c0e-998eaa64c614-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:36:59.312389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:59.312352 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v"] Oct 30 13:36:59.348367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:59.347307 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf"] Oct 30 13:36:59.478799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:59.478746 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:59.586145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:59.586068 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" Oct 30 13:36:59.586145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:59.586089 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk" event={"ID":"36df70c3-5c45-4573-9c0e-998eaa64c614","Type":"ContainerDied","Data":"d861385065f00a37faea55d24ef5a835fa4da53e04af8eff9d805a5a82193494"} Oct 30 13:36:59.586412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:36:59.586152 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d861385065f00a37faea55d24ef5a835fa4da53e04af8eff9d805a5a82193494" Oct 30 13:37:00.562182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:00.561801 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:01.614283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:01.614251 5049 generic.go:356] "Generic (PLEG): container finished" podID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerID="6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a" exitCode=0 Oct 30 13:37:01.614761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:01.614301 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" event={"ID":"9a6b69e3-e48f-4eaa-960d-e149e93cd038","Type":"ContainerDied","Data":"6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a"} Oct 30 13:37:01.614841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:01.614804 5049 scope.go:122] "RemoveContainer" containerID="6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a" Oct 30 13:37:01.618565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:01.618536 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:01.618673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:01.618605 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:01.618673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:01.618638 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:02.477689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.477650 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:02.644723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.644683 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" event={"ID":"9a6b69e3-e48f-4eaa-960d-e149e93cd038","Type":"ContainerStarted","Data":"84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5"} Oct 30 13:37:02.647114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.647086 5049 generic.go:356] "Generic (PLEG): container finished" podID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerID="f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e" exitCode=0 Oct 30 13:37:02.647269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.647166 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" event={"ID":"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36","Type":"ContainerDied","Data":"f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e"} Oct 30 13:37:02.647701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.647681 5049 scope.go:122] "RemoveContainer" containerID="f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e" Oct 30 13:37:02.649574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.649549 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:02.649650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.649617 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:02.649689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.649658 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:02.882619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.882585 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:37:02.883446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.883416 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 13:37:02.883601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:02.883473 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 13:37:03.658567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:03.658531 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" event={"ID":"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36","Type":"ContainerStarted","Data":"154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d"} Oct 30 13:37:03.884340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:03.883304 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 13:37:03.884340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:03.883379 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 13:37:03.923406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:03.916438 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:37:04.142354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:04.141753 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:37:04.145470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:04.145435 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 13:37:04.145719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:04.145693 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 13:37:04.270402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:04.269105 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:04.882927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:04.882892 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 13:37:04.883362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:04.882941 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 13:37:04.935951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:04.935918 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:37:05.147234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:05.146702 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 13:37:05.147234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:05.146764 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 13:37:05.192566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:05.192474 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:05.478404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:05.459731 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:05.885781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:05.885737 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 13:37:05.907452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:05.907401 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 13:37:06.135356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:06.135299 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 13:37:06.135640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:06.135604 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 13:37:06.145305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:06.145217 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:06.882690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:06.882642 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 13:37:06.882879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:06.882707 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 13:37:07.136839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:07.136739 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 13:37:07.137469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:07.137434 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 13:37:07.883530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:07.883492 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Oct 30 13:37:07.883732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:07.883547 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Oct 30 13:37:07.976068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:07.975755 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g"] Oct 30 13:37:08.073363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:08.067802 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:08.146779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:08.146686 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 13:37:08.147350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:08.147302 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 13:37:08.280365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:08.279948 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:08.467305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:08.467196 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:09.083210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:09.083173 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:37:09.087747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:09.087718 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:37:09.137750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:09.137712 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 30 13:37:09.138038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:09.138007 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 30 13:37:10.261757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:10.261717 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:10.318559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:10.318524 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:37:10.320607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:10.320582 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:37:12.214770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:12.210911 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:12.976713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:12.976649 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" podUID="ddeca63d-c87d-4204-92f1-d1972cdee454" containerName="payload-processing" containerID="cri-o://d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774" gracePeriod=25 Oct 30 13:37:13.289523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.289494 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:37:13.413771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.413734 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddeca63d-c87d-4204-92f1-d1972cdee454-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddeca63d-c87d-4204-92f1-d1972cdee454-plugins-config-volume\") pod \"ddeca63d-c87d-4204-92f1-d1972cdee454\" (UID: \"ddeca63d-c87d-4204-92f1-d1972cdee454\") " Oct 30 13:37:13.413963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.413934 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddeca63d-c87d-4204-92f1-d1972cdee454-kube-api-access-m92wm\" (UniqueName: \"kubernetes.io/projected/ddeca63d-c87d-4204-92f1-d1972cdee454-kube-api-access-m92wm\") pod \"ddeca63d-c87d-4204-92f1-d1972cdee454\" (UID: \"ddeca63d-c87d-4204-92f1-d1972cdee454\") " Oct 30 13:37:13.414129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.414106 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeca63d-c87d-4204-92f1-d1972cdee454-plugins-config-volume" pod "ddeca63d-c87d-4204-92f1-d1972cdee454" (UID: "ddeca63d-c87d-4204-92f1-d1972cdee454"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:13.414888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.414860 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddeca63d-c87d-4204-92f1-d1972cdee454-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:13.416233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.416211 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeca63d-c87d-4204-92f1-d1972cdee454-kube-api-access-m92wm" pod "ddeca63d-c87d-4204-92f1-d1972cdee454" (UID: "ddeca63d-c87d-4204-92f1-d1972cdee454"). InnerVolumeSpecName "kube-api-access-m92wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:13.516452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.516364 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m92wm\" (UniqueName: \"kubernetes.io/projected/ddeca63d-c87d-4204-92f1-d1972cdee454-kube-api-access-m92wm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:13.940378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.940302 5049 generic.go:356] "Generic (PLEG): container finished" podID="ddeca63d-c87d-4204-92f1-d1972cdee454" containerID="d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774" exitCode=0 Oct 30 13:37:13.940378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.940366 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" Oct 30 13:37:13.940640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.940389 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" event={"ID":"ddeca63d-c87d-4204-92f1-d1972cdee454","Type":"ContainerDied","Data":"d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774"} Oct 30 13:37:13.940640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.940446 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g" event={"ID":"ddeca63d-c87d-4204-92f1-d1972cdee454","Type":"ContainerDied","Data":"f1540b04ebb92840ad73718127616c902609206bdc71c48b9d09b21e0f4472c2"} Oct 30 13:37:13.940640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.940468 5049 scope.go:122] "RemoveContainer" containerID="d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774" Oct 30 13:37:13.956692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.956671 5049 scope.go:122] "RemoveContainer" containerID="d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774" Oct 30 13:37:13.957029 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:37:13.957009 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774\": container with ID starting with d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774 not found: ID does not exist" containerID="d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774" Oct 30 13:37:13.957149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:13.957039 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774"} err="failed to get container status \"d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774\": rpc error: code = NotFound desc = could not find container \"d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774\": container with ID starting with d76937d50b2fd72f75d8c18495023a4e5b4e92fe989c94d15b9cb724ea249774 not found: ID does not exist" Oct 30 13:37:14.014361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.014304 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g"] Oct 30 13:37:14.081164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.081127 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-64dcc59cd8-46h4g"] Oct 30 13:37:14.185107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.185036 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:14.304672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.304639 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw"] Oct 30 13:37:14.485334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.485277 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:14.503996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.503956 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" containerID="cri-o://ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9" gracePeriod=45 Oct 30 13:37:14.504175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.503961 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" containerID="cri-o://7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f" gracePeriod=45 Oct 30 13:37:14.653924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.653878 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:14.819366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.818283 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:14.879553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.879526 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:37:14.883441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.883417 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.963910 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-home\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-home\") pod \"76b965cb-83d4-4c13-a4be-77060bfca0cc\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.963984 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-model-cache\") pod \"76b965cb-83d4-4c13-a4be-77060bfca0cc\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.964087 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b965cb-83d4-4c13-a4be-77060bfca0cc-kube-api-access-wt64m\" (UniqueName: \"kubernetes.io/projected/76b965cb-83d4-4c13-a4be-77060bfca0cc-kube-api-access-wt64m\") pod \"76b965cb-83d4-4c13-a4be-77060bfca0cc\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.964212 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-tmp-dir\") pod \"381f988e-0231-413c-950d-2302ce2058c2\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.964270 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-dshm\") pod \"76b965cb-83d4-4c13-a4be-77060bfca0cc\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.964374 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76b965cb-83d4-4c13-a4be-77060bfca0cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/76b965cb-83d4-4c13-a4be-77060bfca0cc-tls-certs\") pod \"76b965cb-83d4-4c13-a4be-77060bfca0cc\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.964415 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-home\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-home\") pod \"381f988e-0231-413c-950d-2302ce2058c2\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.964459 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-tmp-dir\") pod \"76b965cb-83d4-4c13-a4be-77060bfca0cc\" (UID: \"76b965cb-83d4-4c13-a4be-77060bfca0cc\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.964530 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381f988e-0231-413c-950d-2302ce2058c2-kube-api-access-wtbf9\" (UniqueName: \"kubernetes.io/projected/381f988e-0231-413c-950d-2302ce2058c2-kube-api-access-wtbf9\") pod \"381f988e-0231-413c-950d-2302ce2058c2\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.964582 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-model-cache\") pod \"381f988e-0231-413c-950d-2302ce2058c2\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.964632 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-dshm\") pod \"381f988e-0231-413c-950d-2302ce2058c2\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " Oct 30 13:37:14.976344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.964673 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381f988e-0231-413c-950d-2302ce2058c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/381f988e-0231-413c-950d-2302ce2058c2-tls-certs\") pod \"381f988e-0231-413c-950d-2302ce2058c2\" (UID: \"381f988e-0231-413c-950d-2302ce2058c2\") " Oct 30 13:37:15.007344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:14.997364 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-tmp-dir" pod "381f988e-0231-413c-950d-2302ce2058c2" (UID: "381f988e-0231-413c-950d-2302ce2058c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:15.007344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.002267 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-home" pod "381f988e-0231-413c-950d-2302ce2058c2" (UID: "381f988e-0231-413c-950d-2302ce2058c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:15.007344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.002381 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-model-cache" pod "381f988e-0231-413c-950d-2302ce2058c2" (UID: "381f988e-0231-413c-950d-2302ce2058c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:15.015041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.014837 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-home" pod "76b965cb-83d4-4c13-a4be-77060bfca0cc" (UID: "76b965cb-83d4-4c13-a4be-77060bfca0cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:15.019377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.015241 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-tmp-dir" pod "76b965cb-83d4-4c13-a4be-77060bfca0cc" (UID: "76b965cb-83d4-4c13-a4be-77060bfca0cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:15.042343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.019570 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-model-cache" pod "76b965cb-83d4-4c13-a4be-77060bfca0cc" (UID: "76b965cb-83d4-4c13-a4be-77060bfca0cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:15.042343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.029774 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-dshm" pod "381f988e-0231-413c-950d-2302ce2058c2" (UID: "381f988e-0231-413c-950d-2302ce2058c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:15.072343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.070308 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.072343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.070393 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.072343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.070410 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.072343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.070424 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.072343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.070437 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.072343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.070450 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.072343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.070462 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381f988e-0231-413c-950d-2302ce2058c2-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.101261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.100415 5049 generic.go:356] "Generic (PLEG): container finished" podID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerID="ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9" exitCode=0 Oct 30 13:37:15.101261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.100640 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" event={"ID":"76b965cb-83d4-4c13-a4be-77060bfca0cc","Type":"ContainerDied","Data":"ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9"} Oct 30 13:37:15.101261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.100680 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" event={"ID":"76b965cb-83d4-4c13-a4be-77060bfca0cc","Type":"ContainerDied","Data":"a96ee9a530b5eca1f143d379894adcd73186265fe0bcc6c862fd0dba4a3cfeb6"} Oct 30 13:37:15.101261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.100702 5049 scope.go:122] "RemoveContainer" containerID="ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9" Oct 30 13:37:15.101261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.100829 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v" Oct 30 13:37:15.106831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.106472 5049 generic.go:356] "Generic (PLEG): container finished" podID="381f988e-0231-413c-950d-2302ce2058c2" containerID="7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f" exitCode=0 Oct 30 13:37:15.106831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.106541 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" event={"ID":"381f988e-0231-413c-950d-2302ce2058c2","Type":"ContainerDied","Data":"7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f"} Oct 30 13:37:15.106831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.106574 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" event={"ID":"381f988e-0231-413c-950d-2302ce2058c2","Type":"ContainerDied","Data":"cb3e5f385836edf91debcff20e750afb215aa0c15be2b078a494bf94e239a61b"} Oct 30 13:37:15.106831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.106639 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" Oct 30 13:37:15.113094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.110665 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381f988e-0231-413c-950d-2302ce2058c2-kube-api-access-wtbf9" pod "381f988e-0231-413c-950d-2302ce2058c2" (UID: "381f988e-0231-413c-950d-2302ce2058c2"). InnerVolumeSpecName "kube-api-access-wtbf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:15.113094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.110810 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b965cb-83d4-4c13-a4be-77060bfca0cc-tls-certs" pod "76b965cb-83d4-4c13-a4be-77060bfca0cc" (UID: "76b965cb-83d4-4c13-a4be-77060bfca0cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:37:15.113094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.110908 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381f988e-0231-413c-950d-2302ce2058c2-tls-certs" pod "381f988e-0231-413c-950d-2302ce2058c2" (UID: "381f988e-0231-413c-950d-2302ce2058c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:37:15.113596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.113556 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b965cb-83d4-4c13-a4be-77060bfca0cc-kube-api-access-wt64m" pod "76b965cb-83d4-4c13-a4be-77060bfca0cc" (UID: "76b965cb-83d4-4c13-a4be-77060bfca0cc"). InnerVolumeSpecName "kube-api-access-wt64m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:15.122829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.122790 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-dshm" pod "76b965cb-83d4-4c13-a4be-77060bfca0cc" (UID: "76b965cb-83d4-4c13-a4be-77060bfca0cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:15.174341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.174291 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtbf9\" (UniqueName: \"kubernetes.io/projected/381f988e-0231-413c-950d-2302ce2058c2-kube-api-access-wtbf9\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.174520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.174346 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381f988e-0231-413c-950d-2302ce2058c2-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.174520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.174364 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt64m\" (UniqueName: \"kubernetes.io/projected/76b965cb-83d4-4c13-a4be-77060bfca0cc-kube-api-access-wt64m\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.174520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.174379 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b965cb-83d4-4c13-a4be-77060bfca0cc-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.174520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.174402 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b965cb-83d4-4c13-a4be-77060bfca0cc-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.204579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.200676 5049 scope.go:122] "RemoveContainer" containerID="ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9" Oct 30 13:37:15.204579 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:37:15.201369 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9\": container with ID starting with ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9 not found: ID does not exist" containerID="ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9" Oct 30 13:37:15.204579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.201414 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9"} err="failed to get container status \"ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9\": rpc error: code = NotFound desc = could not find container \"ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9\": container with ID starting with ae29e4ee7d95dd534088fcbfabd434f64b8ff41e014befcf6091305f4c6b60d9 not found: ID does not exist" Oct 30 13:37:15.204579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.201451 5049 scope.go:122] "RemoveContainer" containerID="7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f" Oct 30 13:37:15.220692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.220668 5049 scope.go:122] "RemoveContainer" containerID="7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f" Oct 30 13:37:15.221192 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:37:15.221166 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f\": container with ID starting with 7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f not found: ID does not exist" containerID="7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f" Oct 30 13:37:15.221388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.221362 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f"} err="failed to get container status \"7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f\": rpc error: code = NotFound desc = could not find container \"7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f\": container with ID starting with 7dd3f74995ee446a26696320e151b9e3274385a26700631beef3d4f1a8ab208f not found: ID does not exist" Oct 30 13:37:15.341921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.341884 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddeca63d-c87d-4204-92f1-d1972cdee454" path="/var/lib/kubelet/pods/ddeca63d-c87d-4204-92f1-d1972cdee454/volumes" Oct 30 13:37:15.449738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.449652 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v"] Oct 30 13:37:15.458047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.458014 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-2-w4-d7e62b-kserve-6c485787c6-kh69v"] Oct 30 13:37:15.478003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.477967 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf"] Oct 30 13:37:15.482970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.482942 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf"] Oct 30 13:37:15.770508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:15.770424 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/premium-w4-d7e62b-kserve-5cb796764d-2zjgf" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:37:16.081245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:16.081207 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:17.338430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:17.338397 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381f988e-0231-413c-950d-2302ce2058c2" path="/var/lib/kubelet/pods/381f988e-0231-413c-950d-2302ce2058c2/volumes" Oct 30 13:37:17.338934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:17.338919 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" path="/var/lib/kubelet/pods/76b965cb-83d4-4c13-a4be-77060bfca0cc/volumes" Oct 30 13:37:18.077584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:18.077502 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:18.234679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:18.234643 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:18.394205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:18.394117 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:19.307349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:19.306383 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" podUID="5857c9db-264a-4342-b143-de98a1498f02" containerName="payload-processing" containerID="cri-o://ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a" gracePeriod=25 Oct 30 13:37:19.631629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:19.631585 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:37:19.752714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:19.752674 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5857c9db-264a-4342-b143-de98a1498f02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5857c9db-264a-4342-b143-de98a1498f02-plugins-config-volume\") pod \"5857c9db-264a-4342-b143-de98a1498f02\" (UID: \"5857c9db-264a-4342-b143-de98a1498f02\") " Oct 30 13:37:19.752904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:19.752749 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5857c9db-264a-4342-b143-de98a1498f02-kube-api-access-ggt85\" (UniqueName: \"kubernetes.io/projected/5857c9db-264a-4342-b143-de98a1498f02-kube-api-access-ggt85\") pod \"5857c9db-264a-4342-b143-de98a1498f02\" (UID: \"5857c9db-264a-4342-b143-de98a1498f02\") " Oct 30 13:37:19.753118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:19.753092 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5857c9db-264a-4342-b143-de98a1498f02-plugins-config-volume" pod "5857c9db-264a-4342-b143-de98a1498f02" (UID: "5857c9db-264a-4342-b143-de98a1498f02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:19.753529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:19.753498 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5857c9db-264a-4342-b143-de98a1498f02-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:19.755071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:19.755045 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5857c9db-264a-4342-b143-de98a1498f02-kube-api-access-ggt85" pod "5857c9db-264a-4342-b143-de98a1498f02" (UID: "5857c9db-264a-4342-b143-de98a1498f02"). InnerVolumeSpecName "kube-api-access-ggt85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:19.854776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:19.854694 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggt85\" (UniqueName: \"kubernetes.io/projected/5857c9db-264a-4342-b143-de98a1498f02-kube-api-access-ggt85\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:20.044764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:20.044728 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:20.178365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:20.178263 5049 generic.go:356] "Generic (PLEG): container finished" podID="5857c9db-264a-4342-b143-de98a1498f02" containerID="ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a" exitCode=0 Oct 30 13:37:20.178365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:20.178310 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" Oct 30 13:37:20.178568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:20.178315 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" event={"ID":"5857c9db-264a-4342-b143-de98a1498f02","Type":"ContainerDied","Data":"ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a"} Oct 30 13:37:20.178568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:20.178494 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw" event={"ID":"5857c9db-264a-4342-b143-de98a1498f02","Type":"ContainerDied","Data":"b246ac4232d0a6342de56c8e9e7fab9554b10cf13abc5b788b20f8e33bac43ae"} Oct 30 13:37:20.178568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:20.178533 5049 scope.go:122] "RemoveContainer" containerID="ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a" Oct 30 13:37:20.193146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:20.193127 5049 scope.go:122] "RemoveContainer" containerID="ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a" Oct 30 13:37:20.193512 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:37:20.193489 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a\": container with ID starting with ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a not found: ID does not exist" containerID="ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a" Oct 30 13:37:20.193590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:20.193520 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a"} err="failed to get container status \"ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a\": rpc error: code = NotFound desc = could not find container \"ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a\": container with ID starting with ad6a8859bf8eec0e2c21d16b36d18884a3fe5e5fb5c44ebe02059f31d396654a not found: ID does not exist" Oct 30 13:37:20.235181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:20.235143 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw"] Oct 30 13:37:20.244682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:20.244650 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-85fcb56998-tvvxw"] Oct 30 13:37:21.339798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:21.339764 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5857c9db-264a-4342-b143-de98a1498f02" path="/var/lib/kubelet/pods/5857c9db-264a-4342-b143-de98a1498f02/volumes" Oct 30 13:37:21.356575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:21.356545 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:21.572152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:21.572113 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:22.644866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:22.644795 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:24.041889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:24.041854 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:24.255222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:24.255188 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:25.418765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:25.418504 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:26.679586 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:37:26.679546 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:37:27.297707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:27.297668 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:29.068634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:29.068599 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:29.519045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:29.518964 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:31.001057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:31.000981 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:32.807278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:32.807224 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:34.608593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:34.608444 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:36.526251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:36.526211 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:36.689781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:36.689749 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:36.809817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:36.809786 5049 scope.go:122] "RemoveContainer" containerID="4e67821c9ffaf97dba269bea0c8bc4eed606b421a7eae56a3a24be7ab2b7395e" Oct 30 13:37:36.822742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:36.822721 5049 scope.go:122] "RemoveContainer" containerID="ffed352db45683e2cdd96aa9b4d60c8169fab78c70b7f8a27cb7294e24425de3" Oct 30 13:37:36.836021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:36.835996 5049 scope.go:122] "RemoveContainer" containerID="3daba02c4f0f91e2d637091db35046eaeefe586660ad756895b12a5894e167be" Oct 30 13:37:36.846758 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:36.846733 5049 scope.go:122] "RemoveContainer" containerID="3dab0f8620dcb8f9236e909308f10e4cf5064b3a5d37660834f67a6e9678b94f" Oct 30 13:37:36.857574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:36.857554 5049 scope.go:122] "RemoveContainer" containerID="19a683d592ac06aaf5f05333c46c1e3fa1aae2d1be75a2a7bc506adac52fffc5" Oct 30 13:37:36.868686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:36.868664 5049 scope.go:122] "RemoveContainer" containerID="3e15c9d0eddb32119dc7638ec0451a92f9ad3e934a947e73c3c65b535fa17eb7" Oct 30 13:37:38.125217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:38.125174 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:39.111612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.111574 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:39.271598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.271566 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:39.640392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.640192 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk"] Oct 30 13:37:39.641493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.641466 5049 cpu_manager.go:414] "containerMap: removing container" podUID="36df70c3-5c45-4573-9c0e-998eaa64c614" containerName="revoke-keys" Oct 30 13:37:39.641679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.641654 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36df70c3-5c45-4573-9c0e-998eaa64c614" containerName="revoke-keys" Oct 30 13:37:39.641807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.641744 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5857c9db-264a-4342-b143-de98a1498f02" containerName="payload-processing" Oct 30 13:37:39.641807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.641753 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5857c9db-264a-4342-b143-de98a1498f02" containerName="payload-processing" Oct 30 13:37:39.641807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.641780 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ddeca63d-c87d-4204-92f1-d1972cdee454" containerName="payload-processing" Oct 30 13:37:39.641807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.641788 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddeca63d-c87d-4204-92f1-d1972cdee454" containerName="payload-processing" Oct 30 13:37:39.641807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.641802 5049 cpu_manager.go:414] "containerMap: removing container" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" Oct 30 13:37:39.642094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.641810 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" Oct 30 13:37:39.642094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.641837 5049 cpu_manager.go:414] "containerMap: removing container" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" Oct 30 13:37:39.642094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.641844 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" Oct 30 13:37:39.642094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.642057 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="381f988e-0231-413c-950d-2302ce2058c2" containerName="main" Oct 30 13:37:39.642094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.642073 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b965cb-83d4-4c13-a4be-77060bfca0cc" containerName="main" Oct 30 13:37:39.642094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.642088 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="36df70c3-5c45-4573-9c0e-998eaa64c614" containerName="revoke-keys" Oct 30 13:37:39.642461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.642107 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5857c9db-264a-4342-b143-de98a1498f02" containerName="payload-processing" Oct 30 13:37:39.642461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.642121 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddeca63d-c87d-4204-92f1-d1972cdee454" containerName="payload-processing" Oct 30 13:37:39.649069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.649041 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.664539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.664470 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c54f0f-openshift-default-dockercfg-c25ls\"" Oct 30 13:37:39.682934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.682902 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk"] Oct 30 13:37:39.778412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.749169 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02eb2218-687c-46f4-8627-a831c5f074fe-istiod-ca-cert\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.778412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.749266 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctq6\" (UniqueName: \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-kube-api-access-tctq6\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.778412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.749405 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-envoy\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.778412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.749452 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-data\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.778412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.749536 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-istio-token\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.778412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.749752 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-certs\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.778412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.749877 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-socket\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.778412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.749942 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-credential-socket\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.778412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.749979 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02eb2218-687c-46f4-8627-a831c5f074fe-istio-podinfo\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.778412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.771008 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:39.852349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.852074 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-envoy\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.852349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.852140 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-data\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.852349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.852204 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-istio-token\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.866354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.865428 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-certs\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.866354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.865548 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-socket\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.866354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.865594 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-credential-socket\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.866354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.865628 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02eb2218-687c-46f4-8627-a831c5f074fe-istio-podinfo\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.866354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.865733 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02eb2218-687c-46f4-8627-a831c5f074fe-istiod-ca-cert\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.866354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.865781 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctq6\" (UniqueName: \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-kube-api-access-tctq6\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.872342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.870152 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-socket\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.872342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.870476 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-data\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.872342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.870495 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-certs\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.872342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.870735 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-credential-socket\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.872342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.871119 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02eb2218-687c-46f4-8627-a831c5f074fe-istiod-ca-cert\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.911401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.906440 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02eb2218-687c-46f4-8627-a831c5f074fe-istio-podinfo\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.927759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.927722 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-envoy\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.945918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.945881 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:39.958354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.950109 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctq6\" (UniqueName: \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-kube-api-access-tctq6\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.959044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.959015 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-istio-token\") pod \"e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:39.971596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:39.971563 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:40.108163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.108090 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:40.432967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.429914 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk"] Oct 30 13:37:40.450765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.449433 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx"] Oct 30 13:37:40.470691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.470657 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.502382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.493889 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-fca422-openshift-default-dockercfg-v5q7m\"" Oct 30 13:37:40.523060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.523008 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:40.523250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.523114 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:40.523250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.523162 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:40.560720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.560592 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:40.596564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.591502 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx"] Oct 30 13:37:40.596564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.593449 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-credential-socket\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.596564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.593622 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-data\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.596564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.593688 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-podinfo\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.596564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.593740 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-certs\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.596564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.593801 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-envoy\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.596564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.593893 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istiod-ca-cert\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.596564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.593926 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skc47\" (UniqueName: \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-kube-api-access-skc47\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.596564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.593963 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-socket\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.596564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.593991 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-token\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.695695 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-podinfo\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.695995 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-certs\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.696155 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-envoy\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.696313 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istiod-ca-cert\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.696385 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skc47\" (UniqueName: \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-kube-api-access-skc47\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.696660 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-socket\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.696720 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-token\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.696966 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-credential-socket\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.697290 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-data\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.708950 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-certs\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.709770 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-credential-socket\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.710533 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-socket\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.710841 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-data\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.711879 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istiod-ca-cert\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.714713 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-envoy\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.726383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.726148 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-token\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.727370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.726635 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:40.749703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.743391 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skc47\" (UniqueName: \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-kube-api-access-skc47\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.758540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.757712 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-podinfo\") pod \"e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:40.938553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:40.938512 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:41.316406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.313373 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:41.317102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.317076 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx"] Oct 30 13:37:41.351561 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:37:41.333745 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574a0063_5e8f_448f_b5d2_a2f182fdbc25.slice/crio-6b0bced5f65c4cbe5e7f0ef30ee2e82f466eb6c2d400f39628c1a1c9ba581e8a WatchSource:0}: Error finding container 6b0bced5f65c4cbe5e7f0ef30ee2e82f466eb6c2d400f39628c1a1c9ba581e8a: Status 404 returned error can't find the container with id 6b0bced5f65c4cbe5e7f0ef30ee2e82f466eb6c2d400f39628c1a1c9ba581e8a Oct 30 13:37:41.351561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.337889 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:41.351561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.337999 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:41.351561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.338075 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:37:41.530458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.530411 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" event={"ID":"574a0063-5e8f-448f-b5d2-a2f182fdbc25","Type":"ContainerStarted","Data":"2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8"} Oct 30 13:37:41.530933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.530479 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" event={"ID":"574a0063-5e8f-448f-b5d2-a2f182fdbc25","Type":"ContainerStarted","Data":"6b0bced5f65c4cbe5e7f0ef30ee2e82f466eb6c2d400f39628c1a1c9ba581e8a"} Oct 30 13:37:41.534028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.533990 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" event={"ID":"02eb2218-687c-46f4-8627-a831c5f074fe","Type":"ContainerStarted","Data":"6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177"} Oct 30 13:37:41.534194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.534055 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" event={"ID":"02eb2218-687c-46f4-8627-a831c5f074fe","Type":"ContainerStarted","Data":"ea66e16ece92f6cabc9a238d6b9c69b4a1bd91a8a9b2ceea3a07ff025b8fe216"} Oct 30 13:37:41.575900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.575793 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" podStartSLOduration=1.575778286 podStartE2EDuration="1.575778286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:41.573524876 +0000 UTC m=+2768.388719301" watchObservedRunningTime="2026-10-30 13:37:41.575778286 +0000 UTC m=+2768.390972709" Oct 30 13:37:41.619540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.619486 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" podStartSLOduration=2.619470996 podStartE2EDuration="2.619470996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:41.614604854 +0000 UTC m=+2768.429799276" watchObservedRunningTime="2026-10-30 13:37:41.619470996 +0000 UTC m=+2768.434665419" Oct 30 13:37:41.939427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.939340 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:41.972498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.972464 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:41.978700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:41.978666 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:42.543446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:42.543419 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:42.544616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:42.544584 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:37:42.947740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:42.947649 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:43.242867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:43.242785 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:43.553669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:43.553635 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:43.554843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:43.554820 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:37:44.065147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:44.065100 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:44.277011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:44.276974 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:44.482004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:44.481919 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:44.941042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:44.941004 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:45.143440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:45.143402 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:45.477873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:45.477790 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:45.687539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:45.687506 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:45.868639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:45.868598 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:47.402601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.402548 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:47.465652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.465607 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4"] Oct 30 13:37:47.476617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.476580 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:37:47.479365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.479107 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4"] Oct 30 13:37:47.495981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.494340 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fca422\"" Oct 30 13:37:47.541835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.541797 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qzgh\" (UniqueName: \"kubernetes.io/projected/c1b80bf4-f870-4369-97c8-cb6a416088f3-kube-api-access-2qzgh\") pod \"payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4\" (UID: \"c1b80bf4-f870-4369-97c8-cb6a416088f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:37:47.542267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.542237 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b80bf4-f870-4369-97c8-cb6a416088f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4\" (UID: \"c1b80bf4-f870-4369-97c8-cb6a416088f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:37:47.572369 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.572337 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:47.662403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.650777 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qzgh\" (UniqueName: \"kubernetes.io/projected/c1b80bf4-f870-4369-97c8-cb6a416088f3-kube-api-access-2qzgh\") pod \"payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4\" (UID: \"c1b80bf4-f870-4369-97c8-cb6a416088f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:37:47.662833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.662800 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b80bf4-f870-4369-97c8-cb6a416088f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4\" (UID: \"c1b80bf4-f870-4369-97c8-cb6a416088f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:37:47.685343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.664241 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b80bf4-f870-4369-97c8-cb6a416088f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4\" (UID: \"c1b80bf4-f870-4369-97c8-cb6a416088f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:37:47.693035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.693003 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qzgh\" (UniqueName: \"kubernetes.io/projected/c1b80bf4-f870-4369-97c8-cb6a416088f3-kube-api-access-2qzgh\") pod \"payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4\" (UID: \"c1b80bf4-f870-4369-97c8-cb6a416088f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:37:47.753880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.753843 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:47.804632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:47.804593 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:37:48.199915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:48.199891 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4"] Oct 30 13:37:48.202596 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:37:48.202568 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b80bf4_f870_4369_97c8_cb6a416088f3.slice/crio-f064f643c7fc215ce6f5870c416bb09e1e3941862393eff4a6595ea3036f2277 WatchSource:0}: Error finding container f064f643c7fc215ce6f5870c416bb09e1e3941862393eff4a6595ea3036f2277: Status 404 returned error can't find the container with id f064f643c7fc215ce6f5870c416bb09e1e3941862393eff4a6595ea3036f2277 Oct 30 13:37:48.601795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:48.601752 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" event={"ID":"c1b80bf4-f870-4369-97c8-cb6a416088f3","Type":"ContainerStarted","Data":"d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a"} Oct 30 13:37:48.602255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:48.601817 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" event={"ID":"c1b80bf4-f870-4369-97c8-cb6a416088f3","Type":"ContainerStarted","Data":"f064f643c7fc215ce6f5870c416bb09e1e3941862393eff4a6595ea3036f2277"} Oct 30 13:37:48.602255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:48.601846 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:37:48.647919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:48.647854 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" podStartSLOduration=1.6478341840000001 podStartE2EDuration="1.647834184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:48.646503002 +0000 UTC m=+2775.461697426" watchObservedRunningTime="2026-10-30 13:37:48.647834184 +0000 UTC m=+2775.463028611" Oct 30 13:37:49.652197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:49.652161 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:51.616949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:51.616882 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:53.908986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:53.908954 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:55.732981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.732943 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh"] Oct 30 13:37:55.742280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.742248 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:55.754835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.754805 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4"] Oct 30 13:37:55.758444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.758416 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:37:55.758634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.758458 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:37:55.761523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.761503 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh"] Oct 30 13:37:55.761635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.761606 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:55.782136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.782102 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4"] Oct 30 13:37:55.829487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.829451 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb5ea26-2mxrh\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:55.829714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.829689 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38f94b4-zv8t4\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:55.829762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.829731 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc9l5\" (UniqueName: \"kubernetes.io/projected/9e80a744-135d-48f0-8f51-f7655b2e54a2-kube-api-access-kc9l5\") pod \"maas-api-revoke-sub-e38f94b4-zv8t4\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:55.829893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.829876 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb5ea26-2mxrh\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:55.829950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.829925 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38f94b4-zv8t4\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:55.830070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.830053 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxx29\" (UniqueName: \"kubernetes.io/projected/faa9db65-bae4-4118-8eee-ec357fcee0a1-kube-api-access-gxx29\") pod \"maas-api-revoke-sub-edb5ea26-2mxrh\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:55.931028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.930994 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38f94b4-zv8t4\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:55.931028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.931035 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc9l5\" (UniqueName: \"kubernetes.io/projected/9e80a744-135d-48f0-8f51-f7655b2e54a2-kube-api-access-kc9l5\") pod \"maas-api-revoke-sub-e38f94b4-zv8t4\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:55.931312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.931125 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb5ea26-2mxrh\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:55.931312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.931164 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38f94b4-zv8t4\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:55.931312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.931270 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxx29\" (UniqueName: \"kubernetes.io/projected/faa9db65-bae4-4118-8eee-ec357fcee0a1-kube-api-access-gxx29\") pod \"maas-api-revoke-sub-edb5ea26-2mxrh\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:55.931533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.931398 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb5ea26-2mxrh\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:55.931918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.931890 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38f94b4-zv8t4\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:55.932046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.932028 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb5ea26-2mxrh\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:55.933748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.933719 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb5ea26-2mxrh\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:55.933929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.933908 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38f94b4-zv8t4\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:55.952795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.952771 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxx29\" (UniqueName: \"kubernetes.io/projected/faa9db65-bae4-4118-8eee-ec357fcee0a1-kube-api-access-gxx29\") pod \"maas-api-revoke-sub-edb5ea26-2mxrh\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:55.952980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:55.952957 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc9l5\" (UniqueName: \"kubernetes.io/projected/9e80a744-135d-48f0-8f51-f7655b2e54a2-kube-api-access-kc9l5\") pod \"maas-api-revoke-sub-e38f94b4-zv8t4\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:56.061186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:56.061148 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:56.095397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:56.095370 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:56.285695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:56.285638 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:56.400523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:56.400478 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh"] Oct 30 13:37:56.402790 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:37:56.402763 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa9db65_bae4_4118_8eee_ec357fcee0a1.slice/crio-086c9f4b13e9a9e3c84591df0dc9b0eba5d5ed43b0d72c446fa0049b9afb213b WatchSource:0}: Error finding container 086c9f4b13e9a9e3c84591df0dc9b0eba5d5ed43b0d72c446fa0049b9afb213b: Status 404 returned error can't find the container with id 086c9f4b13e9a9e3c84591df0dc9b0eba5d5ed43b0d72c446fa0049b9afb213b Oct 30 13:37:56.455110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:56.455083 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4"] Oct 30 13:37:56.456368 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:37:56.456304 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e80a744_135d_48f0_8f51_f7655b2e54a2.slice/crio-190a292f33cfbc8ac9f23db950a4786e01cf16c9011f752d7034e858853222e6 WatchSource:0}: Error finding container 190a292f33cfbc8ac9f23db950a4786e01cf16c9011f752d7034e858853222e6: Status 404 returned error can't find the container with id 190a292f33cfbc8ac9f23db950a4786e01cf16c9011f752d7034e858853222e6 Oct 30 13:37:56.679299 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:37:56.679223 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:37:56.736915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:56.736876 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" event={"ID":"faa9db65-bae4-4118-8eee-ec357fcee0a1","Type":"ContainerStarted","Data":"086c9f4b13e9a9e3c84591df0dc9b0eba5d5ed43b0d72c446fa0049b9afb213b"} Oct 30 13:37:56.738451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:56.738411 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" event={"ID":"9e80a744-135d-48f0-8f51-f7655b2e54a2","Type":"ContainerStarted","Data":"190a292f33cfbc8ac9f23db950a4786e01cf16c9011f752d7034e858853222e6"} Oct 30 13:37:57.748500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:57.748419 5049 generic.go:356] "Generic (PLEG): container finished" podID="faa9db65-bae4-4118-8eee-ec357fcee0a1" containerID="549dc5fa949b1354973ab53be109d93b7278a87488a1fbf2f952ad12033a52f5" exitCode=0 Oct 30 13:37:57.749025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:57.748494 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" event={"ID":"faa9db65-bae4-4118-8eee-ec357fcee0a1","Type":"ContainerDied","Data":"549dc5fa949b1354973ab53be109d93b7278a87488a1fbf2f952ad12033a52f5"} Oct 30 13:37:57.750502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:57.750472 5049 generic.go:356] "Generic (PLEG): container finished" podID="9e80a744-135d-48f0-8f51-f7655b2e54a2" containerID="7bf7443e7b81189fdfe17e74f183e29abe5c1bd8a033cab39608f420ffc8a74c" exitCode=0 Oct 30 13:37:57.750617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:57.750509 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" event={"ID":"9e80a744-135d-48f0-8f51-f7655b2e54a2","Type":"ContainerDied","Data":"7bf7443e7b81189fdfe17e74f183e29abe5c1bd8a033cab39608f420ffc8a74c"} Oct 30 13:37:58.284337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:58.284288 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:58.441129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:58.441087 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:58.597777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:58.597734 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:59.200231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.200201 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:59.204490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.204465 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:59.360930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.360897 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-openshift-ca\") pod \"faa9db65-bae4-4118-8eee-ec357fcee0a1\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " Oct 30 13:37:59.361114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.360984 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-openshift-ca\") pod \"9e80a744-135d-48f0-8f51-f7655b2e54a2\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " Oct 30 13:37:59.361188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.361144 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-xks-ca\") pod \"faa9db65-bae4-4118-8eee-ec357fcee0a1\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " Oct 30 13:37:59.361607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.361288 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e80a744-135d-48f0-8f51-f7655b2e54a2-kube-api-access-kc9l5\" (UniqueName: \"kubernetes.io/projected/9e80a744-135d-48f0-8f51-f7655b2e54a2-kube-api-access-kc9l5\") pod \"9e80a744-135d-48f0-8f51-f7655b2e54a2\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " Oct 30 13:37:59.361607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.361342 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-openshift-ca" pod "9e80a744-135d-48f0-8f51-f7655b2e54a2" (UID: "9e80a744-135d-48f0-8f51-f7655b2e54a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:59.361607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.361360 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-openshift-ca" pod "faa9db65-bae4-4118-8eee-ec357fcee0a1" (UID: "faa9db65-bae4-4118-8eee-ec357fcee0a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:59.361607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.361498 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa9db65-bae4-4118-8eee-ec357fcee0a1-kube-api-access-gxx29\" (UniqueName: \"kubernetes.io/projected/faa9db65-bae4-4118-8eee-ec357fcee0a1-kube-api-access-gxx29\") pod \"faa9db65-bae4-4118-8eee-ec357fcee0a1\" (UID: \"faa9db65-bae4-4118-8eee-ec357fcee0a1\") " Oct 30 13:37:59.361901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.361676 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-xks-ca\") pod \"9e80a744-135d-48f0-8f51-f7655b2e54a2\" (UID: \"9e80a744-135d-48f0-8f51-f7655b2e54a2\") " Oct 30 13:37:59.364048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.364021 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e80a744-135d-48f0-8f51-f7655b2e54a2-kube-api-access-kc9l5" pod "9e80a744-135d-48f0-8f51-f7655b2e54a2" (UID: "9e80a744-135d-48f0-8f51-f7655b2e54a2"). InnerVolumeSpecName "kube-api-access-kc9l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:59.364191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.364065 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa9db65-bae4-4118-8eee-ec357fcee0a1-kube-api-access-gxx29" pod "faa9db65-bae4-4118-8eee-ec357fcee0a1" (UID: "faa9db65-bae4-4118-8eee-ec357fcee0a1"). InnerVolumeSpecName "kube-api-access-gxx29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:59.364191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.364093 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-xks-ca" pod "faa9db65-bae4-4118-8eee-ec357fcee0a1" (UID: "faa9db65-bae4-4118-8eee-ec357fcee0a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:37:59.364619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.364593 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-xks-ca" pod "9e80a744-135d-48f0-8f51-f7655b2e54a2" (UID: "9e80a744-135d-48f0-8f51-f7655b2e54a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:37:59.365412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.365391 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.365504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.365418 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.467768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.467727 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc9l5\" (UniqueName: \"kubernetes.io/projected/9e80a744-135d-48f0-8f51-f7655b2e54a2-kube-api-access-kc9l5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.467768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.467764 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxx29\" (UniqueName: \"kubernetes.io/projected/faa9db65-bae4-4118-8eee-ec357fcee0a1-kube-api-access-gxx29\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.467976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.467781 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e80a744-135d-48f0-8f51-f7655b2e54a2-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.467976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.467794 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa9db65-bae4-4118-8eee-ec357fcee0a1-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:37:59.645434 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.645356 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:37:59.819614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.819585 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" Oct 30 13:37:59.819820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.819618 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4" event={"ID":"9e80a744-135d-48f0-8f51-f7655b2e54a2","Type":"ContainerDied","Data":"190a292f33cfbc8ac9f23db950a4786e01cf16c9011f752d7034e858853222e6"} Oct 30 13:37:59.819820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.819709 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="190a292f33cfbc8ac9f23db950a4786e01cf16c9011f752d7034e858853222e6" Oct 30 13:37:59.821162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.821140 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" Oct 30 13:37:59.821276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.821174 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh" event={"ID":"faa9db65-bae4-4118-8eee-ec357fcee0a1","Type":"ContainerDied","Data":"086c9f4b13e9a9e3c84591df0dc9b0eba5d5ed43b0d72c446fa0049b9afb213b"} Oct 30 13:37:59.821276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.821211 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="086c9f4b13e9a9e3c84591df0dc9b0eba5d5ed43b0d72c446fa0049b9afb213b" Oct 30 13:37:59.960625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:37:59.960536 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:00.147346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:00.136716 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:01.248566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:01.248532 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:03.200303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:03.200267 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:04.664517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:04.664481 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:04.880298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:04.880264 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:05.539502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:05.539468 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:06.436127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:06.436087 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:08.348873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:08.348840 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:10.111834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:10.111777 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:12.062986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:12.061598 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:14.024609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:14.024544 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:16.140021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:16.139982 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:18.534973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:18.534872 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:18.773286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:18.773246 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:18.975611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:18.975577 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:20.358366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.358313 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8"] Oct 30 13:38:20.359619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.359596 5049 cpu_manager.go:414] "containerMap: removing container" podUID="faa9db65-bae4-4118-8eee-ec357fcee0a1" containerName="revoke-keys" Oct 30 13:38:20.359691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.359621 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa9db65-bae4-4118-8eee-ec357fcee0a1" containerName="revoke-keys" Oct 30 13:38:20.359691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.359657 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9e80a744-135d-48f0-8f51-f7655b2e54a2" containerName="revoke-keys" Oct 30 13:38:20.359691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.359667 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e80a744-135d-48f0-8f51-f7655b2e54a2" containerName="revoke-keys" Oct 30 13:38:20.359955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.359941 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e80a744-135d-48f0-8f51-f7655b2e54a2" containerName="revoke-keys" Oct 30 13:38:20.359990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.359979 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa9db65-bae4-4118-8eee-ec357fcee0a1" containerName="revoke-keys" Oct 30 13:38:20.365485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.365462 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:20.383296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.383268 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8"] Oct 30 13:38:20.385564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.385541 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:38:20.385702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.385671 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:38:20.428287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.428256 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jshx\" (UniqueName: \"kubernetes.io/projected/9262ea07-b8e4-4951-bc05-4203f9038e78-kube-api-access-2jshx\") pod \"maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:20.428524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.428394 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:20.428684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.428659 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:20.530463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.530426 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jshx\" (UniqueName: \"kubernetes.io/projected/9262ea07-b8e4-4951-bc05-4203f9038e78-kube-api-access-2jshx\") pod \"maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:20.530667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.530547 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:20.530749 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.530738 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:20.531203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.531183 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:20.533160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.533134 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:20.552021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.551989 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jshx\" (UniqueName: \"kubernetes.io/projected/9262ea07-b8e4-4951-bc05-4203f9038e78-kube-api-access-2jshx\") pod \"maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:20.684551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:20.684459 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:21.028080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:21.028053 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8"] Oct 30 13:38:21.035857 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:38:21.035819 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9262ea07_b8e4_4951_bc05_4203f9038e78.slice/crio-074af3e4f79153c5450e185b24da7ca2de65cd79d71d22775d6db8d202abee0e WatchSource:0}: Error finding container 074af3e4f79153c5450e185b24da7ca2de65cd79d71d22775d6db8d202abee0e: Status 404 returned error can't find the container with id 074af3e4f79153c5450e185b24da7ca2de65cd79d71d22775d6db8d202abee0e Oct 30 13:38:21.174745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:21.174688 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:22.057592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:22.057567 5049 generic.go:356] "Generic (PLEG): container finished" podID="9262ea07-b8e4-4951-bc05-4203f9038e78" containerID="f9747a572cf44477dc5217d1d45f5c89bc78c65b0207a83d8bec8e4ebb920748" exitCode=0 Oct 30 13:38:22.057972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:22.057660 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" event={"ID":"9262ea07-b8e4-4951-bc05-4203f9038e78","Type":"ContainerDied","Data":"f9747a572cf44477dc5217d1d45f5c89bc78c65b0207a83d8bec8e4ebb920748"} Oct 30 13:38:22.057972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:22.057723 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" event={"ID":"9262ea07-b8e4-4951-bc05-4203f9038e78","Type":"ContainerStarted","Data":"074af3e4f79153c5450e185b24da7ca2de65cd79d71d22775d6db8d202abee0e"} Oct 30 13:38:22.514557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:22.514520 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:22.701816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:22.701779 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:23.239355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.239309 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:23.661605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.661560 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:23.680160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.680129 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:23.822714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.822677 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:23.858235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.858204 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-openshift-ca\") pod \"9262ea07-b8e4-4951-bc05-4203f9038e78\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " Oct 30 13:38:23.858487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.858370 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-xks-ca\") pod \"9262ea07-b8e4-4951-bc05-4203f9038e78\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " Oct 30 13:38:23.858577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.858485 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9262ea07-b8e4-4951-bc05-4203f9038e78-kube-api-access-2jshx\" (UniqueName: \"kubernetes.io/projected/9262ea07-b8e4-4951-bc05-4203f9038e78-kube-api-access-2jshx\") pod \"9262ea07-b8e4-4951-bc05-4203f9038e78\" (UID: \"9262ea07-b8e4-4951-bc05-4203f9038e78\") " Oct 30 13:38:23.858689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.858661 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-openshift-ca" pod "9262ea07-b8e4-4951-bc05-4203f9038e78" (UID: "9262ea07-b8e4-4951-bc05-4203f9038e78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:23.859225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.859202 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:23.860684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.860659 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-xks-ca" pod "9262ea07-b8e4-4951-bc05-4203f9038e78" (UID: "9262ea07-b8e4-4951-bc05-4203f9038e78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:23.861064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.861039 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9262ea07-b8e4-4951-bc05-4203f9038e78-kube-api-access-2jshx" pod "9262ea07-b8e4-4951-bc05-4203f9038e78" (UID: "9262ea07-b8e4-4951-bc05-4203f9038e78"). InnerVolumeSpecName "kube-api-access-2jshx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:23.961062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.961024 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9262ea07-b8e4-4951-bc05-4203f9038e78-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:23.961062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:23.961062 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jshx\" (UniqueName: \"kubernetes.io/projected/9262ea07-b8e4-4951-bc05-4203f9038e78-kube-api-access-2jshx\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:24.088910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:24.088816 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" Oct 30 13:38:24.089159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:24.089126 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8" event={"ID":"9262ea07-b8e4-4951-bc05-4203f9038e78","Type":"ContainerDied","Data":"074af3e4f79153c5450e185b24da7ca2de65cd79d71d22775d6db8d202abee0e"} Oct 30 13:38:24.089245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:24.089193 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="074af3e4f79153c5450e185b24da7ca2de65cd79d71d22775d6db8d202abee0e" Oct 30 13:38:25.697433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:25.696554 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:26.679916 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:38:26.679872 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:38:27.348569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:27.348537 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:27.556026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:27.555984 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:27.819832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:27.819798 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:29.270551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:29.270516 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:29.526404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:29.491238 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:29.718456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:29.710292 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:30.064242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:30.064194 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:31.878829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:31.878792 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:33.423124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:33.423091 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:33.632926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:33.632892 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:33.800043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:33.799996 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:34.416829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:34.416790 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:34.520673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:34.520637 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8"] Oct 30 13:38:34.530790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:34.530761 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-082748-mzpq8"] Oct 30 13:38:34.569287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:34.569256 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:34.714371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:34.714272 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:35.046400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:35.046368 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:35.337886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:35.337812 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9262ea07-b8e4-4951-bc05-4203f9038e78" path="/var/lib/kubelet/pods/9262ea07-b8e4-4951-bc05-4203f9038e78/volumes" Oct 30 13:38:36.507609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:36.507572 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:36.975658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:36.975629 5049 scope.go:122] "RemoveContainer" containerID="ef2234e12655d5feaff38e75ee909a44076b87a25e6a2bccf71481d6001abb1c" Oct 30 13:38:38.301822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:38.301789 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:38.461358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:38.461303 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:40.247147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:40.247111 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:40.417160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:40.417129 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:41.780221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:41.780189 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:41.937013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:41.936979 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:42.135938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.135900 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c"] Oct 30 13:38:42.136199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.136164 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" containerID="cri-o://a3382d1ff4f9c18b2af8e785505cc39217db7cfe1c3295eefccf791652f237aa" gracePeriod=30 Oct 30 13:38:42.197703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.197669 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:42.376503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.376445 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:42.599744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.599708 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84"] Oct 30 13:38:42.601691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.601653 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9262ea07-b8e4-4951-bc05-4203f9038e78" containerName="revoke-keys" Oct 30 13:38:42.601691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.601689 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9262ea07-b8e4-4951-bc05-4203f9038e78" containerName="revoke-keys" Oct 30 13:38:42.601983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.601962 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9262ea07-b8e4-4951-bc05-4203f9038e78" containerName="revoke-keys" Oct 30 13:38:42.608942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.608909 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:42.626651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.626621 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:38:42.627203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.627180 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:38:42.630915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.630889 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84"] Oct 30 13:38:42.657386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.657151 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:42.657386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.657307 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8hv\" (UniqueName: \"kubernetes.io/projected/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-kube-api-access-tj8hv\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:42.657619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.657545 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:42.763891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.763844 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:42.764113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.764082 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:42.764188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.764169 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8hv\" (UniqueName: \"kubernetes.io/projected/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-kube-api-access-tj8hv\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:42.765438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.765408 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:42.771763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.771736 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:42.789465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.789439 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8hv\" (UniqueName: \"kubernetes.io/projected/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-kube-api-access-tj8hv\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:42.962734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:42.962649 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:43.294134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:43.294108 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84"] Oct 30 13:38:43.296600 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:38:43.296573 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23f30ce_9d5c_4e7b_a13d_ce215943bbd2.slice/crio-7aeb37b228c084637fcf90fc6e6c77ea8b7b80a0d90b9263d19c9dee0afc064d WatchSource:0}: Error finding container 7aeb37b228c084637fcf90fc6e6c77ea8b7b80a0d90b9263d19c9dee0afc064d: Status 404 returned error can't find the container with id 7aeb37b228c084637fcf90fc6e6c77ea8b7b80a0d90b9263d19c9dee0afc064d Oct 30 13:38:43.346565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:43.346525 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" event={"ID":"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2","Type":"ContainerStarted","Data":"7aeb37b228c084637fcf90fc6e6c77ea8b7b80a0d90b9263d19c9dee0afc064d"} Oct 30 13:38:44.357199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:44.357164 5049 generic.go:356] "Generic (PLEG): container finished" podID="b23f30ce-9d5c-4e7b-a13d-ce215943bbd2" containerID="627f8df0fee96d17b62771ad2a0a55d914316f29d8376af8844dbf61f2cdb8d7" exitCode=0 Oct 30 13:38:44.357636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:44.357255 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" event={"ID":"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2","Type":"ContainerDied","Data":"627f8df0fee96d17b62771ad2a0a55d914316f29d8376af8844dbf61f2cdb8d7"} Oct 30 13:38:44.709685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:44.706096 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:45.568836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:45.568810 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:45.610471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:45.610437 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-openshift-ca\") pod \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " Oct 30 13:38:45.610756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:45.610729 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-kube-api-access-tj8hv\" (UniqueName: \"kubernetes.io/projected/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-kube-api-access-tj8hv\") pod \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " Oct 30 13:38:45.611510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:45.610850 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-xks-ca\") pod \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\" (UID: \"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2\") " Oct 30 13:38:45.611510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:45.610954 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-openshift-ca" pod "b23f30ce-9d5c-4e7b-a13d-ce215943bbd2" (UID: "b23f30ce-9d5c-4e7b-a13d-ce215943bbd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:45.611813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:45.611641 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:45.613743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:45.613708 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-kube-api-access-tj8hv" pod "b23f30ce-9d5c-4e7b-a13d-ce215943bbd2" (UID: "b23f30ce-9d5c-4e7b-a13d-ce215943bbd2"). InnerVolumeSpecName "kube-api-access-tj8hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:45.613835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:45.613800 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-xks-ca" pod "b23f30ce-9d5c-4e7b-a13d-ce215943bbd2" (UID: "b23f30ce-9d5c-4e7b-a13d-ce215943bbd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:45.712969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:45.712935 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj8hv\" (UniqueName: \"kubernetes.io/projected/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-kube-api-access-tj8hv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:45.712969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:45.712968 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:46.375713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:46.375680 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" event={"ID":"b23f30ce-9d5c-4e7b-a13d-ce215943bbd2","Type":"ContainerDied","Data":"7aeb37b228c084637fcf90fc6e6c77ea8b7b80a0d90b9263d19c9dee0afc064d"} Oct 30 13:38:46.375999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:46.375723 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84" Oct 30 13:38:46.375999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:46.375727 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aeb37b228c084637fcf90fc6e6c77ea8b7b80a0d90b9263d19c9dee0afc064d" Oct 30 13:38:46.866813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:46.859649 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:47.394498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.394463 5049 generic.go:356] "Generic (PLEG): container finished" podID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerID="a3382d1ff4f9c18b2af8e785505cc39217db7cfe1c3295eefccf791652f237aa" exitCode=0 Oct 30 13:38:47.394705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.394543 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" event={"ID":"ef45dd97-be25-4314-9a4b-08d5041c2dd5","Type":"ContainerDied","Data":"a3382d1ff4f9c18b2af8e785505cc39217db7cfe1c3295eefccf791652f237aa"} Oct 30 13:38:47.482845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.482817 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:38:47.552968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.552926 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-envoy\") pod \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " Oct 30 13:38:47.553222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.553036 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-token\" (UniqueName: \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-token\") pod \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " Oct 30 13:38:47.553222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.553066 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-certs\") pod \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " Oct 30 13:38:47.553367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.553231 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-kube-api-access-fbhrn\" (UniqueName: \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-kube-api-access-fbhrn\") pod \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " Oct 30 13:38:47.554605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.554573 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-podinfo\") pod \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " Oct 30 13:38:47.554746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.554633 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-credential-socket\") pod \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " Oct 30 13:38:47.554746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.554691 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-data\") pod \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " Oct 30 13:38:47.554877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.554773 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-socket\") pod \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " Oct 30 13:38:47.554877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.554815 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istiod-ca-cert\") pod \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\" (UID: \"ef45dd97-be25-4314-9a4b-08d5041c2dd5\") " Oct 30 13:38:47.556136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.556102 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istiod-ca-cert" pod "ef45dd97-be25-4314-9a4b-08d5041c2dd5" (UID: "ef45dd97-be25-4314-9a4b-08d5041c2dd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:47.557934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.556986 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-socket" pod "ef45dd97-be25-4314-9a4b-08d5041c2dd5" (UID: "ef45dd97-be25-4314-9a4b-08d5041c2dd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.557934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.557193 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-data" pod "ef45dd97-be25-4314-9a4b-08d5041c2dd5" (UID: "ef45dd97-be25-4314-9a4b-08d5041c2dd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.557934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.557856 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-certs" pod "ef45dd97-be25-4314-9a4b-08d5041c2dd5" (UID: "ef45dd97-be25-4314-9a4b-08d5041c2dd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.557934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.557913 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-credential-socket" pod "ef45dd97-be25-4314-9a4b-08d5041c2dd5" (UID: "ef45dd97-be25-4314-9a4b-08d5041c2dd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.560701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.560656 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-envoy" pod "ef45dd97-be25-4314-9a4b-08d5041c2dd5" (UID: "ef45dd97-be25-4314-9a4b-08d5041c2dd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.561145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.561119 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-token" pod "ef45dd97-be25-4314-9a4b-08d5041c2dd5" (UID: "ef45dd97-be25-4314-9a4b-08d5041c2dd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:47.563798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.563770 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-kube-api-access-fbhrn" pod "ef45dd97-be25-4314-9a4b-08d5041c2dd5" (UID: "ef45dd97-be25-4314-9a4b-08d5041c2dd5"). InnerVolumeSpecName "kube-api-access-fbhrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:47.565625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.565602 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-podinfo" pod "ef45dd97-be25-4314-9a4b-08d5041c2dd5" (UID: "ef45dd97-be25-4314-9a4b-08d5041c2dd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:38:47.656803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.656766 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.656803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.656796 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.656803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.656809 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.657053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.656821 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.657053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.656830 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.657053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.656839 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.657053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.656846 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.657053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.656854 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef45dd97-be25-4314-9a4b-08d5041c2dd5-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.657053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:47.656862 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbhrn\" (UniqueName: \"kubernetes.io/projected/ef45dd97-be25-4314-9a4b-08d5041c2dd5-kube-api-access-fbhrn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.033180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:48.033132 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr"] Oct 30 13:38:48.040514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:48.040487 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07129f75-fdpkr"] Oct 30 13:38:48.405626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:48.405592 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" Oct 30 13:38:48.405626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:48.405607 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c" event={"ID":"ef45dd97-be25-4314-9a4b-08d5041c2dd5","Type":"ContainerDied","Data":"d737c437158424b571c115dd81c398e09f1fde63103d24738d9eb16558fa3ac3"} Oct 30 13:38:48.406000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:48.405687 5049 scope.go:122] "RemoveContainer" containerID="a3382d1ff4f9c18b2af8e785505cc39217db7cfe1c3295eefccf791652f237aa" Oct 30 13:38:48.456351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:48.456299 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c"] Oct 30 13:38:48.466097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:48.466062 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-mmv8c"] Oct 30 13:38:48.711867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:48.711744 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:49.339694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:49.339660 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad73dbe-40ec-4e07-a567-0d3b811063b6" path="/var/lib/kubelet/pods/9ad73dbe-40ec-4e07-a567-0d3b811063b6/volumes" Oct 30 13:38:49.340199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:49.340183 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" path="/var/lib/kubelet/pods/ef45dd97-be25-4314-9a4b-08d5041c2dd5/volumes" Oct 30 13:38:50.034757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:50.034720 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj"] Oct 30 13:38:50.044313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:50.044280 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ebfa0e-4wzxj"] Oct 30 13:38:50.511167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:50.511132 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:50.663981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:50.663947 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:51.030578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:51.030543 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz"] Oct 30 13:38:51.042842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:51.042804 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30230832-pp4fz"] Oct 30 13:38:51.339042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:51.339007 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c279e4-7510-4613-865d-88b96986a27c" path="/var/lib/kubelet/pods/99c279e4-7510-4613-865d-88b96986a27c/volumes" Oct 30 13:38:51.339572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:51.339556 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d981c28b-a864-4825-841f-df7978785cc8" path="/var/lib/kubelet/pods/d981c28b-a864-4825-841f-df7978785cc8/volumes" Oct 30 13:38:51.885725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:51.885689 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:52.114607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:52.114553 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:53.119002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:53.118965 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:54.599002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:54.598966 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr"] Oct 30 13:38:54.599520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:54.599218 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" podUID="58abbd14-8811-435e-b0f2-0826e0ba32eb" containerName="maas-api" containerID="cri-o://0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291" gracePeriod=30 Oct 30 13:38:54.726931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:54.726885 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:55.234863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.234813 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:38:55.322058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.322021 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:55.359894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.352160 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-maas-api-tls\") pod \"58abbd14-8811-435e-b0f2-0826e0ba32eb\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " Oct 30 13:38:55.359894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.352392 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-metrics-certs\") pod \"58abbd14-8811-435e-b0f2-0826e0ba32eb\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " Oct 30 13:38:55.359894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.352505 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58abbd14-8811-435e-b0f2-0826e0ba32eb-kube-api-access-kf87g\" (UniqueName: \"kubernetes.io/projected/58abbd14-8811-435e-b0f2-0826e0ba32eb-kube-api-access-kf87g\") pod \"58abbd14-8811-435e-b0f2-0826e0ba32eb\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " Oct 30 13:38:55.359894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.352719 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58abbd14-8811-435e-b0f2-0826e0ba32eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58abbd14-8811-435e-b0f2-0826e0ba32eb-openshift-service-ca\") pod \"58abbd14-8811-435e-b0f2-0826e0ba32eb\" (UID: \"58abbd14-8811-435e-b0f2-0826e0ba32eb\") " Oct 30 13:38:55.370607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.370567 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58abbd14-8811-435e-b0f2-0826e0ba32eb-openshift-service-ca" pod "58abbd14-8811-435e-b0f2-0826e0ba32eb" (UID: "58abbd14-8811-435e-b0f2-0826e0ba32eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:55.380997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.380944 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58abbd14-8811-435e-b0f2-0826e0ba32eb-kube-api-access-kf87g" pod "58abbd14-8811-435e-b0f2-0826e0ba32eb" (UID: "58abbd14-8811-435e-b0f2-0826e0ba32eb"). InnerVolumeSpecName "kube-api-access-kf87g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:55.384052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.384023 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-metrics-certs" pod "58abbd14-8811-435e-b0f2-0826e0ba32eb" (UID: "58abbd14-8811-435e-b0f2-0826e0ba32eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:55.384400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.384377 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-maas-api-tls" pod "58abbd14-8811-435e-b0f2-0826e0ba32eb" (UID: "58abbd14-8811-435e-b0f2-0826e0ba32eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:55.455577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.455541 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58abbd14-8811-435e-b0f2-0826e0ba32eb-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:55.455577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.455570 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:55.455577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.455582 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58abbd14-8811-435e-b0f2-0826e0ba32eb-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:55.455878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.455591 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf87g\" (UniqueName: \"kubernetes.io/projected/58abbd14-8811-435e-b0f2-0826e0ba32eb-kube-api-access-kf87g\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:38:55.462873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.462840 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk"] Oct 30 13:38:55.468042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.468012 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d7e62b-cqrsk"] Oct 30 13:38:55.502831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.502795 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:55.635225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.635107 5049 generic.go:356] "Generic (PLEG): container finished" podID="58abbd14-8811-435e-b0f2-0826e0ba32eb" containerID="0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291" exitCode=0 Oct 30 13:38:55.636293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.636259 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" event={"ID":"58abbd14-8811-435e-b0f2-0826e0ba32eb","Type":"ContainerDied","Data":"0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291"} Oct 30 13:38:55.636592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.636554 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" event={"ID":"58abbd14-8811-435e-b0f2-0826e0ba32eb","Type":"ContainerDied","Data":"ff9d28be0077dbd30e86fcea5f51b77c40db0155da8ea45d449b2f24925f463a"} Oct 30 13:38:55.636739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.636724 5049 scope.go:122] "RemoveContainer" containerID="0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291" Oct 30 13:38:55.646467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.645485 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr" Oct 30 13:38:55.710538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.710313 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:55.741305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.740991 5049 scope.go:122] "RemoveContainer" containerID="0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291" Oct 30 13:38:55.741674 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:38:55.741576 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291\": container with ID starting with 0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291 not found: ID does not exist" containerID="0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291" Oct 30 13:38:55.741674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.741616 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291"} err="failed to get container status \"0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291\": rpc error: code = NotFound desc = could not find container \"0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291\": container with ID starting with 0885673d0ad6a6a5d7ee794ec1e8ba33a47e33e820c613b678d9edb6d0ad9291 not found: ID does not exist" Oct 30 13:38:55.752582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.751471 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr"] Oct 30 13:38:55.758203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.758172 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d7e62b-5879646646-s6fxr"] Oct 30 13:38:55.971172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:55.971090 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:56.697525 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:38:56.697491 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:38:57.345193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:57.345162 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36df70c3-5c45-4573-9c0e-998eaa64c614" path="/var/lib/kubelet/pods/36df70c3-5c45-4573-9c0e-998eaa64c614/volumes" Oct 30 13:38:57.345734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:57.345719 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58abbd14-8811-435e-b0f2-0826e0ba32eb" path="/var/lib/kubelet/pods/58abbd14-8811-435e-b0f2-0826e0ba32eb/volumes" Oct 30 13:38:57.673048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:57.672958 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:58.030402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:58.009840 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:58.055341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:58.051346 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw"] Oct 30 13:38:58.059821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:58.058446 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfd24a8-ft7vw"] Oct 30 13:38:58.289233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:58.289137 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:58.581363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:58.581262 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:58.742637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:58.742598 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84"] Oct 30 13:38:58.769993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:58.769925 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b254d-lrp84"] Oct 30 13:38:58.774532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:58.774392 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:58.943061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:58.942962 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:59.274773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:59.274681 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:59.353839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:59.353798 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b23f30ce-9d5c-4e7b-a13d-ce215943bbd2" path="/var/lib/kubelet/pods/b23f30ce-9d5c-4e7b-a13d-ce215943bbd2/volumes" Oct 30 13:38:59.354457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:38:59.354429 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b558bfdf-310b-44ec-80f3-8a966f418f25" path="/var/lib/kubelet/pods/b558bfdf-310b-44ec-80f3-8a966f418f25/volumes" Oct 30 13:39:00.043748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:00.043710 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4"] Oct 30 13:39:00.048370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:00.048313 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338efa68-twqv4"] Oct 30 13:39:00.835232 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:00.835193 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:01.039050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:01.039004 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl"] Oct 30 13:39:01.045759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:01.045727 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a75b2a72-gxdpl"] Oct 30 13:39:01.352336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:01.352291 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08156865-c43b-4b9e-9231-be3922b57868" path="/var/lib/kubelet/pods/08156865-c43b-4b9e-9231-be3922b57868/volumes" Oct 30 13:39:01.353049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:01.353029 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a2207cc-2aa7-4552-83f6-ed428807529e" path="/var/lib/kubelet/pods/3a2207cc-2aa7-4552-83f6-ed428807529e/volumes" Oct 30 13:39:02.880333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:02.880275 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:03.035826 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:03.035791 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:03.154057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:03.153922 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb"] Oct 30 13:39:03.154543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:03.154499 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" containerID="cri-o://818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26" gracePeriod=30 Oct 30 13:39:03.235401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:03.235360 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:03.442391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:03.436495 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:05.252359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:05.251394 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:06.177021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.176990 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:06.333613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.333579 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:06.467627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.467590 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx"] Oct 30 13:39:06.508607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.468015 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" containerID="cri-o://84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5" gracePeriod=30 Oct 30 13:39:06.515956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.515923 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:06.680921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.680639 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:06.963649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.963609 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl"] Oct 30 13:39:06.964462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.964444 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b23f30ce-9d5c-4e7b-a13d-ce215943bbd2" containerName="revoke-keys" Oct 30 13:39:06.964462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.964462 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23f30ce-9d5c-4e7b-a13d-ce215943bbd2" containerName="revoke-keys" Oct 30 13:39:06.964571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.964495 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" Oct 30 13:39:06.964571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.964500 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" Oct 30 13:39:06.964571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.964518 5049 cpu_manager.go:414] "containerMap: removing container" podUID="58abbd14-8811-435e-b0f2-0826e0ba32eb" containerName="maas-api" Oct 30 13:39:06.964571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.964523 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58abbd14-8811-435e-b0f2-0826e0ba32eb" containerName="maas-api" Oct 30 13:39:06.964712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.964651 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b23f30ce-9d5c-4e7b-a13d-ce215943bbd2" containerName="revoke-keys" Oct 30 13:39:06.964712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.964665 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef45dd97-be25-4314-9a4b-08d5041c2dd5" containerName="istio-proxy" Oct 30 13:39:06.964712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.964674 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="58abbd14-8811-435e-b0f2-0826e0ba32eb" containerName="maas-api" Oct 30 13:39:06.968619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.968591 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:06.984138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.984111 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e3ebf5-openshift-default-dockercfg-tm22r\"" Oct 30 13:39:06.998174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:06.998149 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7"] Oct 30 13:39:07.003371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.003312 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.021768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.021734 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl"] Oct 30 13:39:07.054008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.053662 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:39:07.055284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.054603 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:39:07.058602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.058574 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7"] Oct 30 13:39:07.086346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.081965 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:07.095125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095099 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-istio-token\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.095360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095230 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.095360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095281 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4jh\" (UniqueName: \"kubernetes.io/projected/82f36460-2c00-40c2-a296-9c2fd336dcd4-kube-api-access-vj4jh\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.095528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095367 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.095572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095535 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-certs\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.095616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095568 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a56f54-da40-419f-947b-34164430f3cc-istiod-ca-cert\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.095736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095718 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a56f54-da40-419f-947b-34164430f3cc-istio-podinfo\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.095805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095753 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjpqs\" (UniqueName: \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-kube-api-access-kjpqs\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.095805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095788 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-socket\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.095960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095933 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-data\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.096017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.095993 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-envoy\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.096098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.096082 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-credential-socket\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.220555 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-envoy\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.220824 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-credential-socket\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.220992 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-istio-token\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.221280 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.221346 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4jh\" (UniqueName: \"kubernetes.io/projected/82f36460-2c00-40c2-a296-9c2fd336dcd4-kube-api-access-vj4jh\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.221395 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.221694 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-certs\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.221733 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a56f54-da40-419f-947b-34164430f3cc-istiod-ca-cert\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.222027 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a56f54-da40-419f-947b-34164430f3cc-istio-podinfo\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.222148 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjpqs\" (UniqueName: \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-kube-api-access-kjpqs\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.222182 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-socket\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.222450 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-data\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.225402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.224966 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-certs\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.226263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.225674 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.226354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.226268 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a56f54-da40-419f-947b-34164430f3cc-istiod-ca-cert\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.246341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.227083 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-socket\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.246598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.228623 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-credential-socket\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.246764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.238878 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-data\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.254125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.254101 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.254819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.254784 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a56f54-da40-419f-947b-34164430f3cc-istio-podinfo\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.267349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.265562 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-envoy\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.281287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.281218 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:07.282382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.282286 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjpqs\" (UniqueName: \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-kube-api-access-kjpqs\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.285355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.284790 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-istio-token\") pod \"e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.289610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.289575 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4jh\" (UniqueName: \"kubernetes.io/projected/82f36460-2c00-40c2-a296-9c2fd336dcd4-kube-api-access-vj4jh\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.315054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.314977 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:07.350842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.350616 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:07.741830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.741798 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7"] Oct 30 13:39:07.742488 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:39:07.742465 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f36460_2c00_40c2_a296_9c2fd336dcd4.slice/crio-b840d576690b2c0788f77d34213fad393f88b72cc1fdee8c10eaedd2fcf543be WatchSource:0}: Error finding container b840d576690b2c0788f77d34213fad393f88b72cc1fdee8c10eaedd2fcf543be: Status 404 returned error can't find the container with id b840d576690b2c0788f77d34213fad393f88b72cc1fdee8c10eaedd2fcf543be Oct 30 13:39:07.764242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.764213 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl"] Oct 30 13:39:07.768631 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:39:07.768601 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a56f54_da40_419f_947b_34164430f3cc.slice/crio-9c4441acf2f8ed9d2b2fa502c92891ee25526161b2b09017f1ab78b2a987b383 WatchSource:0}: Error finding container 9c4441acf2f8ed9d2b2fa502c92891ee25526161b2b09017f1ab78b2a987b383: Status 404 returned error can't find the container with id 9c4441acf2f8ed9d2b2fa502c92891ee25526161b2b09017f1ab78b2a987b383 Oct 30 13:39:07.772492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.772460 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:07.772610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.772546 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:07.772610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.772590 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:07.802535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.802501 5049 generic.go:356] "Generic (PLEG): container finished" podID="02eb2218-687c-46f4-8627-a831c5f074fe" containerID="6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177" exitCode=0 Oct 30 13:39:07.802638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.802571 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" event={"ID":"02eb2218-687c-46f4-8627-a831c5f074fe","Type":"ContainerDied","Data":"6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177"} Oct 30 13:39:07.803055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.803039 5049 scope.go:122] "RemoveContainer" containerID="6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177" Oct 30 13:39:07.804054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.804029 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" event={"ID":"22a56f54-da40-419f-947b-34164430f3cc","Type":"ContainerStarted","Data":"9c4441acf2f8ed9d2b2fa502c92891ee25526161b2b09017f1ab78b2a987b383"} Oct 30 13:39:07.805170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.805136 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:07.805276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.805224 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:07.805391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.805272 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:07.806120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:07.805667 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" event={"ID":"82f36460-2c00-40c2-a296-9c2fd336dcd4","Type":"ContainerStarted","Data":"b840d576690b2c0788f77d34213fad393f88b72cc1fdee8c10eaedd2fcf543be"} Oct 30 13:39:08.502343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.502287 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:39:08.664166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664133 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9df030fd-edbd-4631-b7e5-292564e90d08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df030fd-edbd-4631-b7e5-292564e90d08-istio-podinfo\") pod \"9df030fd-edbd-4631-b7e5-292564e90d08\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " Oct 30 13:39:08.664375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664188 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-data\") pod \"9df030fd-edbd-4631-b7e5-292564e90d08\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " Oct 30 13:39:08.664375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664247 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-envoy\") pod \"9df030fd-edbd-4631-b7e5-292564e90d08\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " Oct 30 13:39:08.664516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664399 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-kube-api-access-vn7kc\" (UniqueName: \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-kube-api-access-vn7kc\") pod \"9df030fd-edbd-4631-b7e5-292564e90d08\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " Oct 30 13:39:08.664516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664426 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df030fd-edbd-4631-b7e5-292564e90d08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df030fd-edbd-4631-b7e5-292564e90d08-istiod-ca-cert\") pod \"9df030fd-edbd-4631-b7e5-292564e90d08\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " Oct 30 13:39:08.664516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664499 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-credential-socket\") pod \"9df030fd-edbd-4631-b7e5-292564e90d08\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " Oct 30 13:39:08.664680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664604 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-data" pod "9df030fd-edbd-4631-b7e5-292564e90d08" (UID: "9df030fd-edbd-4631-b7e5-292564e90d08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:08.664680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664656 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-certs\") pod \"9df030fd-edbd-4631-b7e5-292564e90d08\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " Oct 30 13:39:08.664807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664715 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-istio-token\" (UniqueName: \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-istio-token\") pod \"9df030fd-edbd-4631-b7e5-292564e90d08\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " Oct 30 13:39:08.664807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664747 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-socket\") pod \"9df030fd-edbd-4631-b7e5-292564e90d08\" (UID: \"9df030fd-edbd-4631-b7e5-292564e90d08\") " Oct 30 13:39:08.664946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664910 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-certs" pod "9df030fd-edbd-4631-b7e5-292564e90d08" (UID: "9df030fd-edbd-4631-b7e5-292564e90d08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:08.665019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.664942 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df030fd-edbd-4631-b7e5-292564e90d08-istiod-ca-cert" pod "9df030fd-edbd-4631-b7e5-292564e90d08" (UID: "9df030fd-edbd-4631-b7e5-292564e90d08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:08.665183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.665165 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-socket" pod "9df030fd-edbd-4631-b7e5-292564e90d08" (UID: "9df030fd-edbd-4631-b7e5-292564e90d08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:08.665256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.665149 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-credential-socket" pod "9df030fd-edbd-4631-b7e5-292564e90d08" (UID: "9df030fd-edbd-4631-b7e5-292564e90d08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:08.665563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.665537 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.665656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.665570 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.665656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.665585 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.665656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.665599 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df030fd-edbd-4631-b7e5-292564e90d08-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.665656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.665613 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.666735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.666708 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-kube-api-access-vn7kc" pod "9df030fd-edbd-4631-b7e5-292564e90d08" (UID: "9df030fd-edbd-4631-b7e5-292564e90d08"). InnerVolumeSpecName "kube-api-access-vn7kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:08.667010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.666991 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-envoy" pod "9df030fd-edbd-4631-b7e5-292564e90d08" (UID: "9df030fd-edbd-4631-b7e5-292564e90d08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:08.667062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.667011 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-istio-token" pod "9df030fd-edbd-4631-b7e5-292564e90d08" (UID: "9df030fd-edbd-4631-b7e5-292564e90d08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:08.667198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.667174 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9df030fd-edbd-4631-b7e5-292564e90d08-istio-podinfo" pod "9df030fd-edbd-4631-b7e5-292564e90d08" (UID: "9df030fd-edbd-4631-b7e5-292564e90d08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:08.771221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.771178 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn7kc\" (UniqueName: \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-kube-api-access-vn7kc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.771221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.771220 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df030fd-edbd-4631-b7e5-292564e90d08-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.771548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.771235 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df030fd-edbd-4631-b7e5-292564e90d08-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.771548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.771248 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df030fd-edbd-4631-b7e5-292564e90d08-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:08.848938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.848904 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" event={"ID":"02eb2218-687c-46f4-8627-a831c5f074fe","Type":"ContainerStarted","Data":"b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb"} Oct 30 13:39:08.851035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.851002 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" event={"ID":"22a56f54-da40-419f-947b-34164430f3cc","Type":"ContainerStarted","Data":"51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e"} Oct 30 13:39:08.852946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.852922 5049 generic.go:356] "Generic (PLEG): container finished" podID="82f36460-2c00-40c2-a296-9c2fd336dcd4" containerID="0d9530422e3475df369b11a6cc7eec5071f83e643d712b8a27a330f0d8d5e943" exitCode=0 Oct 30 13:39:08.853082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.852983 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" event={"ID":"82f36460-2c00-40c2-a296-9c2fd336dcd4","Type":"ContainerDied","Data":"0d9530422e3475df369b11a6cc7eec5071f83e643d712b8a27a330f0d8d5e943"} Oct 30 13:39:08.855162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.855138 5049 generic.go:356] "Generic (PLEG): container finished" podID="9df030fd-edbd-4631-b7e5-292564e90d08" containerID="818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26" exitCode=0 Oct 30 13:39:08.855308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.855180 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" Oct 30 13:39:08.855308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.855209 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" event={"ID":"9df030fd-edbd-4631-b7e5-292564e90d08","Type":"ContainerDied","Data":"818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26"} Oct 30 13:39:08.855308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.855254 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb" event={"ID":"9df030fd-edbd-4631-b7e5-292564e90d08","Type":"ContainerDied","Data":"ff144ee82740e6e7575a8750f294ef3d3d80d715e5b90872d35e0d0bda1cb997"} Oct 30 13:39:08.855308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.855286 5049 scope.go:122] "RemoveContainer" containerID="818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26" Oct 30 13:39:08.868706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.868689 5049 scope.go:122] "RemoveContainer" containerID="73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d" Oct 30 13:39:08.879796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.879774 5049 scope.go:122] "RemoveContainer" containerID="818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26" Oct 30 13:39:08.880073 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:39:08.880054 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26\": container with ID starting with 818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26 not found: ID does not exist" containerID="818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26" Oct 30 13:39:08.880118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.880082 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26"} err="failed to get container status \"818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26\": rpc error: code = NotFound desc = could not find container \"818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26\": container with ID starting with 818965167e5c1abafc1da2ccc14866e65813dfc5022b2d1f5de2cfd351e24e26 not found: ID does not exist" Oct 30 13:39:08.880118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.880100 5049 scope.go:122] "RemoveContainer" containerID="73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d" Oct 30 13:39:08.880376 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:39:08.880358 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d\": container with ID starting with 73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d not found: ID does not exist" containerID="73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d" Oct 30 13:39:08.880470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.880376 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d"} err="failed to get container status \"73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d\": rpc error: code = NotFound desc = could not find container \"73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d\": container with ID starting with 73fa5df3d0968d1305349b80600e49e1648450acc31aa58e6169f5979008392d not found: ID does not exist" Oct 30 13:39:08.938402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.937784 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb"] Oct 30 13:39:08.944665 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.944620 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xgkfb"] Oct 30 13:39:08.972185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:08.972154 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:39:09.040356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.034905 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" podStartSLOduration=3.034881605 podStartE2EDuration="3.034881605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:09.012641593 +0000 UTC m=+2855.827836017" watchObservedRunningTime="2026-10-30 13:39:09.034881605 +0000 UTC m=+2855.850076031" Oct 30 13:39:09.086512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.086475 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:39:09.086660 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.086547 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:39:09.327346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.327291 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:09.351564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.351516 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" path="/var/lib/kubelet/pods/9df030fd-edbd-4631-b7e5-292564e90d08/volumes" Oct 30 13:39:09.352272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.352251 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:09.401203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.401164 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:09.879307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.878853 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:09.879903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.879877 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:39:09.976314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.974637 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 30 13:39:09.976314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:09.974700 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 30 13:39:10.183488 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.183461 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:10.298343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.298295 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-openshift-ca\") pod \"82f36460-2c00-40c2-a296-9c2fd336dcd4\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " Oct 30 13:39:10.298545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.298474 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-xks-ca\") pod \"82f36460-2c00-40c2-a296-9c2fd336dcd4\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " Oct 30 13:39:10.298661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.298642 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f36460-2c00-40c2-a296-9c2fd336dcd4-kube-api-access-vj4jh\" (UniqueName: \"kubernetes.io/projected/82f36460-2c00-40c2-a296-9c2fd336dcd4-kube-api-access-vj4jh\") pod \"82f36460-2c00-40c2-a296-9c2fd336dcd4\" (UID: \"82f36460-2c00-40c2-a296-9c2fd336dcd4\") " Oct 30 13:39:10.299085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.299060 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-openshift-ca" pod "82f36460-2c00-40c2-a296-9c2fd336dcd4" (UID: "82f36460-2c00-40c2-a296-9c2fd336dcd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:10.299549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.299510 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:10.301492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.301466 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f36460-2c00-40c2-a296-9c2fd336dcd4-kube-api-access-vj4jh" pod "82f36460-2c00-40c2-a296-9c2fd336dcd4" (UID: "82f36460-2c00-40c2-a296-9c2fd336dcd4"). InnerVolumeSpecName "kube-api-access-vj4jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:10.301572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.301556 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-xks-ca" pod "82f36460-2c00-40c2-a296-9c2fd336dcd4" (UID: "82f36460-2c00-40c2-a296-9c2fd336dcd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:10.401007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.400922 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f36460-2c00-40c2-a296-9c2fd336dcd4-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:10.401007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.400951 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj4jh\" (UniqueName: \"kubernetes.io/projected/82f36460-2c00-40c2-a296-9c2fd336dcd4-kube-api-access-vj4jh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:10.886356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.886297 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" Oct 30 13:39:10.886737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.886362 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7" event={"ID":"82f36460-2c00-40c2-a296-9c2fd336dcd4","Type":"ContainerDied","Data":"b840d576690b2c0788f77d34213fad393f88b72cc1fdee8c10eaedd2fcf543be"} Oct 30 13:39:10.886737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.886416 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b840d576690b2c0788f77d34213fad393f88b72cc1fdee8c10eaedd2fcf543be" Oct 30 13:39:10.973223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.973181 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 30 13:39:10.973432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:10.973251 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 30 13:39:11.540348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.539875 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:11.717353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.713725 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:11.879545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.879511 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:11.890411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.890385 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:39:11.898546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.898520 5049 generic.go:356] "Generic (PLEG): container finished" podID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerID="84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5" exitCode=0 Oct 30 13:39:11.898653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.898571 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" Oct 30 13:39:11.898653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.898568 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" event={"ID":"9a6b69e3-e48f-4eaa-960d-e149e93cd038","Type":"ContainerDied","Data":"84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5"} Oct 30 13:39:11.898753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.898670 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx" event={"ID":"9a6b69e3-e48f-4eaa-960d-e149e93cd038","Type":"ContainerDied","Data":"07f8cade572edbcb19f6ac2b8169d10dabb3146a1c918fe8236ba2061e84be80"} Oct 30 13:39:11.898753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.898700 5049 scope.go:122] "RemoveContainer" containerID="84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5" Oct 30 13:39:11.942315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.941807 5049 scope.go:122] "RemoveContainer" containerID="6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a" Oct 30 13:39:11.955591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.955565 5049 scope.go:122] "RemoveContainer" containerID="84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5" Oct 30 13:39:11.955934 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:39:11.955910 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5\": container with ID starting with 84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5 not found: ID does not exist" containerID="84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5" Oct 30 13:39:11.956000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.955947 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5"} err="failed to get container status \"84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5\": rpc error: code = NotFound desc = could not find container \"84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5\": container with ID starting with 84987a0183bf3a24210698f3cab42cdf25b98b5ddfd197c766f27bc63d2106f5 not found: ID does not exist" Oct 30 13:39:11.956000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.955974 5049 scope.go:122] "RemoveContainer" containerID="6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a" Oct 30 13:39:11.956262 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:39:11.956237 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a\": container with ID starting with 6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a not found: ID does not exist" containerID="6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a" Oct 30 13:39:11.956359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.956273 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a"} err="failed to get container status \"6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a\": rpc error: code = NotFound desc = could not find container \"6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a\": container with ID starting with 6db446df4def2bb695055dfb8e25f63bb7aeb701f866c769035a364f8c0ddd9a not found: ID does not exist" Oct 30 13:39:11.972500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.972477 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 30 13:39:11.972606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:11.972521 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.054391 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istiod-ca-cert\") pod \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.054454 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-token\") pod \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.054513 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-envoy\") pod \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.054731 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-data\") pod \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.054801 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-socket\") pod \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.054845 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-podinfo\") pod \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.054877 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-kube-api-access-zv94f\" (UniqueName: \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-kube-api-access-zv94f\") pod \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.054912 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-certs\") pod \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.055038 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-credential-socket\") pod \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\" (UID: \"9a6b69e3-e48f-4eaa-960d-e149e93cd038\") " Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.055467 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-socket" pod "9a6b69e3-e48f-4eaa-960d-e149e93cd038" (UID: "9a6b69e3-e48f-4eaa-960d-e149e93cd038"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.055487 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-credential-socket" pod "9a6b69e3-e48f-4eaa-960d-e149e93cd038" (UID: "9a6b69e3-e48f-4eaa-960d-e149e93cd038"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.056169 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.056182 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istiod-ca-cert" pod "9a6b69e3-e48f-4eaa-960d-e149e93cd038" (UID: "9a6b69e3-e48f-4eaa-960d-e149e93cd038"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.056190 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.056574 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-certs" pod "9a6b69e3-e48f-4eaa-960d-e149e93cd038" (UID: "9a6b69e3-e48f-4eaa-960d-e149e93cd038"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:12.057940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.056791 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-data" pod "9a6b69e3-e48f-4eaa-960d-e149e93cd038" (UID: "9a6b69e3-e48f-4eaa-960d-e149e93cd038"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:12.059636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.059598 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-token" pod "9a6b69e3-e48f-4eaa-960d-e149e93cd038" (UID: "9a6b69e3-e48f-4eaa-960d-e149e93cd038"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:12.061412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.061388 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-kube-api-access-zv94f" pod "9a6b69e3-e48f-4eaa-960d-e149e93cd038" (UID: "9a6b69e3-e48f-4eaa-960d-e149e93cd038"). InnerVolumeSpecName "kube-api-access-zv94f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:12.061801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.061770 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-podinfo" pod "9a6b69e3-e48f-4eaa-960d-e149e93cd038" (UID: "9a6b69e3-e48f-4eaa-960d-e149e93cd038"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:39:12.062713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.062680 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-envoy" pod "9a6b69e3-e48f-4eaa-960d-e149e93cd038" (UID: "9a6b69e3-e48f-4eaa-960d-e149e93cd038"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:39:12.157994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.157896 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:12.157994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.157926 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:12.157994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.157936 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv94f\" (UniqueName: \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-kube-api-access-zv94f\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:12.157994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.157946 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:12.157994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.157957 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:12.157994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.157965 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:12.157994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.157972 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b69e3-e48f-4eaa-960d-e149e93cd038-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:12.282004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.281965 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx"] Oct 30 13:39:12.291517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.291487 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b254d-openshift-default-8684f4bc5f-4dqxx"] Oct 30 13:39:12.543772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.543744 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:39:12.972571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.972483 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 30 13:39:12.973065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:12.972562 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 30 13:39:13.058969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.058930 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:13.295375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.295345 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:13.357829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.357793 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" path="/var/lib/kubelet/pods/9a6b69e3-e48f-4eaa-960d-e149e93cd038/volumes" Oct 30 13:39:13.473738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.473232 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:13.604710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.604626 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd"] Oct 30 13:39:13.614345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.605565 5049 cpu_manager.go:414] "containerMap: removing container" podUID="82f36460-2c00-40c2-a296-9c2fd336dcd4" containerName="revoke-keys" Oct 30 13:39:13.614581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.614557 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f36460-2c00-40c2-a296-9c2fd336dcd4" containerName="revoke-keys" Oct 30 13:39:13.614700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.614687 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" Oct 30 13:39:13.614857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.614844 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" Oct 30 13:39:13.614967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.614954 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" Oct 30 13:39:13.615046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.615035 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" Oct 30 13:39:13.615159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.615148 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" Oct 30 13:39:13.615239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.615229 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" Oct 30 13:39:13.615390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.615376 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" Oct 30 13:39:13.615500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.615489 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" Oct 30 13:39:13.615846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.615830 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f36460-2c00-40c2-a296-9c2fd336dcd4" containerName="revoke-keys" Oct 30 13:39:13.615965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.615954 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" Oct 30 13:39:13.616051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.616042 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" Oct 30 13:39:13.616139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.616129 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6b69e3-e48f-4eaa-960d-e149e93cd038" containerName="istio-proxy" Oct 30 13:39:13.616239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.616229 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df030fd-edbd-4631-b7e5-292564e90d08" containerName="istio-proxy" Oct 30 13:39:13.646225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.646182 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:13.670089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.670054 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd"] Oct 30 13:39:13.670282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.670166 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:39:13.688456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.685492 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e3ebf5\"" Oct 30 13:39:13.688456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.686161 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e3ebf5-dockercfg-ng46w\"" Oct 30 13:39:13.733371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.713710 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd\" (UID: \"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:39:13.733371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.713827 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5j2j\" (UniqueName: \"kubernetes.io/projected/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-kube-api-access-l5j2j\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd\" (UID: \"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:39:13.774345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:13.774276 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8"] Oct 30 13:39:14.009395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.004392 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 30 13:39:14.009395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.004461 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 30 13:39:14.059214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.059172 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd\" (UID: \"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:39:14.059417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.059363 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5j2j\" (UniqueName: \"kubernetes.io/projected/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-kube-api-access-l5j2j\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd\" (UID: \"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:39:14.066942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.064695 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd\" (UID: \"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:39:14.077167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.077137 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8"] Oct 30 13:39:14.077315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.077293 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:39:14.088133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.087403 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5j2j\" (UniqueName: \"kubernetes.io/projected/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-kube-api-access-l5j2j\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd\" (UID: \"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:39:14.263775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.263683 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww2qn\" (UniqueName: \"kubernetes.io/projected/135ac821-40f8-46bf-a3c4-f133dd61053e-kube-api-access-ww2qn\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8\" (UID: \"135ac821-40f8-46bf-a3c4-f133dd61053e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:39:14.263935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.263800 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135ac821-40f8-46bf-a3c4-f133dd61053e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8\" (UID: \"135ac821-40f8-46bf-a3c4-f133dd61053e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:39:14.297703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.297676 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:39:14.366039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.366005 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww2qn\" (UniqueName: \"kubernetes.io/projected/135ac821-40f8-46bf-a3c4-f133dd61053e-kube-api-access-ww2qn\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8\" (UID: \"135ac821-40f8-46bf-a3c4-f133dd61053e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:39:14.366196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.366113 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135ac821-40f8-46bf-a3c4-f133dd61053e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8\" (UID: \"135ac821-40f8-46bf-a3c4-f133dd61053e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:39:14.367080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.367051 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135ac821-40f8-46bf-a3c4-f133dd61053e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8\" (UID: \"135ac821-40f8-46bf-a3c4-f133dd61053e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:39:14.388984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.388952 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww2qn\" (UniqueName: \"kubernetes.io/projected/135ac821-40f8-46bf-a3c4-f133dd61053e-kube-api-access-ww2qn\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8\" (UID: \"135ac821-40f8-46bf-a3c4-f133dd61053e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:39:14.444611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.444384 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:39:14.642341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.641943 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:14.649213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.649184 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd"] Oct 30 13:39:14.828061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.828033 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8"] Oct 30 13:39:14.829870 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:39:14.829827 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135ac821_40f8_46bf_a3c4_f133dd61053e.slice/crio-2a8c883e6c080423ff70cc69ace52ad684cd90dc98441500249e515b9c5e83f2 WatchSource:0}: Error finding container 2a8c883e6c080423ff70cc69ace52ad684cd90dc98441500249e515b9c5e83f2: Status 404 returned error can't find the container with id 2a8c883e6c080423ff70cc69ace52ad684cd90dc98441500249e515b9c5e83f2 Oct 30 13:39:14.977479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.977445 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:39:14.978427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:14.978398 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:39:15.102045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:15.101948 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" event={"ID":"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9","Type":"ContainerStarted","Data":"8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949"} Oct 30 13:39:15.102045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:15.102019 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" event={"ID":"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9","Type":"ContainerStarted","Data":"48d0539f43229e93d2ea8ae046911c330046c9591627eee6070556ee738a7f8d"} Oct 30 13:39:15.102632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:15.102555 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:39:15.106399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:15.106300 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" event={"ID":"135ac821-40f8-46bf-a3c4-f133dd61053e","Type":"ContainerStarted","Data":"92a1a787a7fa06a6f5955a9d7bbced3f17e856cf19e65ccc44da6ae5162a08bf"} Oct 30 13:39:15.106744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:15.106715 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:39:15.106865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:15.106767 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" event={"ID":"135ac821-40f8-46bf-a3c4-f133dd61053e","Type":"ContainerStarted","Data":"2a8c883e6c080423ff70cc69ace52ad684cd90dc98441500249e515b9c5e83f2"} Oct 30 13:39:15.139058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:15.138997 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" podStartSLOduration=2.138981486 podStartE2EDuration="2.138981486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:15.138832125 +0000 UTC m=+2861.954026549" watchObservedRunningTime="2026-10-30 13:39:15.138981486 +0000 UTC m=+2861.954175910" Oct 30 13:39:15.180412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:15.180363 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" podStartSLOduration=2.18035037 podStartE2EDuration="2.18035037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:15.175333739 +0000 UTC m=+2861.990528201" watchObservedRunningTime="2026-10-30 13:39:15.18035037 +0000 UTC m=+2861.995544789" Oct 30 13:39:17.083355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:17.083290 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:17.623611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:17.623579 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:17.769038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:17.769007 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:19.540548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:19.540514 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:21.407010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:21.406972 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:23.224593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:23.224551 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:24.613253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:24.613221 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:24.771800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:24.771763 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:24.948451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:24.948416 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:25.130172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:25.130137 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:25.298514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:25.298472 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:26.118123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:26.118092 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:39:26.118667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:26.118178 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:39:26.679172 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:39:26.679122 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:39:26.904666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:26.904623 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:27.054299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.054265 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:27.283400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.283366 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds"] Oct 30 13:39:27.289192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.289147 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.305495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.305418 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8ee35e-openshift-default-dockercfg-27hz2\"" Oct 30 13:39:27.324978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.324944 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds"] Oct 30 13:39:27.497347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.458654 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.497347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.458791 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4b7d\" (UniqueName: \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-kube-api-access-v4b7d\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.497347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.459040 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ab940e-48f1-4439-8842-e7bbba2b45b7-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.497347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.459195 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.497347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.459221 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.497347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.459304 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.497347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.459356 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.497347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.459395 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.497347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.459454 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.509353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.504119 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:27.571398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.571196 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.571398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.571283 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.571398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.571356 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.571741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.571432 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4b7d\" (UniqueName: \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-kube-api-access-v4b7d\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.571741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.571643 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ab940e-48f1-4439-8842-e7bbba2b45b7-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.571863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.571745 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.571863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.571771 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.571863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.571843 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.572048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.571876 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.582361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.572350 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.582361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.573217 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ab940e-48f1-4439-8842-e7bbba2b45b7-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.582361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.573489 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.582361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.574065 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.582361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.579988 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.586352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.583788 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.624760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.624256 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4b7d\" (UniqueName: \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-kube-api-access-v4b7d\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.629951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.629876 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.632057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.632027 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:27.878450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.878365 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:27.913718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:27.913685 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:28.266668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:28.266633 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds"] Oct 30 13:39:28.267737 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:39:28.267709 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ab940e_48f1_4439_8842_e7bbba2b45b7.slice/crio-97fbaf835ce8768d14ad19bcecea9f146897293da4e4e51d59a0b3dc70d54389 WatchSource:0}: Error finding container 97fbaf835ce8768d14ad19bcecea9f146897293da4e4e51d59a0b3dc70d54389: Status 404 returned error can't find the container with id 97fbaf835ce8768d14ad19bcecea9f146897293da4e4e51d59a0b3dc70d54389 Oct 30 13:39:28.270816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:28.270782 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:28.270913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:28.270849 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:28.270913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:28.270881 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:28.358968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:28.358939 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" event={"ID":"46ab940e-48f1-4439-8842-e7bbba2b45b7","Type":"ContainerStarted","Data":"30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07"} Oct 30 13:39:28.359466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:28.358995 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" event={"ID":"46ab940e-48f1-4439-8842-e7bbba2b45b7","Type":"ContainerStarted","Data":"97fbaf835ce8768d14ad19bcecea9f146897293da4e4e51d59a0b3dc70d54389"} Oct 30 13:39:28.403919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:28.403854 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" podStartSLOduration=1.403834177 podStartE2EDuration="1.403834177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:28.399817914 +0000 UTC m=+2875.215012371" watchObservedRunningTime="2026-10-30 13:39:28.403834177 +0000 UTC m=+2875.219028601" Oct 30 13:39:28.836341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:28.834282 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:28.914444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:28.914412 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:29.823154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:29.823104 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:29.922355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:29.922307 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:30.376933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:30.376902 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:30.377962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:30.377942 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:39:30.901448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:30.901407 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:31.078343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:31.066186 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:32.563246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.563207 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:32.621176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.621138 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n"] Oct 30 13:39:32.637570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.637520 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26"] Oct 30 13:39:32.640388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.637842 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:39:32.662571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.662548 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8ee35e\"" Oct 30 13:39:32.669656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.669634 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:39:32.670643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.670615 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n"] Oct 30 13:39:32.672948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.672921 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d20ced-fe9a-4750-881d-835f9a94da0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n\" (UID: \"73d20ced-fe9a-4750-881d-835f9a94da0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:39:32.673179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.673157 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pttv\" (UniqueName: \"kubernetes.io/projected/73d20ced-fe9a-4750-881d-835f9a94da0b-kube-api-access-7pttv\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n\" (UID: \"73d20ced-fe9a-4750-881d-835f9a94da0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:39:32.678417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.678389 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26"] Oct 30 13:39:32.738708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.738669 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:32.774828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.774788 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pttv\" (UniqueName: \"kubernetes.io/projected/73d20ced-fe9a-4750-881d-835f9a94da0b-kube-api-access-7pttv\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n\" (UID: \"73d20ced-fe9a-4750-881d-835f9a94da0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:39:32.774988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.774943 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9ks\" (UniqueName: \"kubernetes.io/projected/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-kube-api-access-vw9ks\") pod \"payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26\" (UID: \"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:39:32.775059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.775043 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26\" (UID: \"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:39:32.775188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.775174 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d20ced-fe9a-4750-881d-835f9a94da0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n\" (UID: \"73d20ced-fe9a-4750-881d-835f9a94da0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:39:32.775916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.775898 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d20ced-fe9a-4750-881d-835f9a94da0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n\" (UID: \"73d20ced-fe9a-4750-881d-835f9a94da0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:39:32.803053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.803016 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pttv\" (UniqueName: \"kubernetes.io/projected/73d20ced-fe9a-4750-881d-835f9a94da0b-kube-api-access-7pttv\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n\" (UID: \"73d20ced-fe9a-4750-881d-835f9a94da0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:39:32.881055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.880957 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9ks\" (UniqueName: \"kubernetes.io/projected/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-kube-api-access-vw9ks\") pod \"payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26\" (UID: \"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:39:32.881251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.881068 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26\" (UID: \"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:39:32.881879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.881857 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26\" (UID: \"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:39:32.937344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.927202 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9ks\" (UniqueName: \"kubernetes.io/projected/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-kube-api-access-vw9ks\") pod \"payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26\" (UID: \"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:39:32.983346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.980615 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:39:32.998903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:32.990533 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:39:33.515435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:33.473406 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:33.581539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:33.581509 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7"] Oct 30 13:39:33.601664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:33.600442 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8b77b0-544n7"] Oct 30 13:39:33.627785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:33.627654 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:33.739574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:33.739538 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n"] Oct 30 13:39:33.740637 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:39:33.740609 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d20ced_fe9a_4750_881d_835f9a94da0b.slice/crio-d6ca507c3d5f1fe1b623ce2d09f982493ef7458bd8e6b5060029218ff20b64e9 WatchSource:0}: Error finding container d6ca507c3d5f1fe1b623ce2d09f982493ef7458bd8e6b5060029218ff20b64e9: Status 404 returned error can't find the container with id d6ca507c3d5f1fe1b623ce2d09f982493ef7458bd8e6b5060029218ff20b64e9 Oct 30 13:39:33.780039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:33.780015 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26"] Oct 30 13:39:33.782737 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:39:33.782712 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8c604c_6b7e_42d8_8d5f_2d0104ba84f5.slice/crio-2493980310cb4f4afb697d4d55b351e722075b2f491eda0b88a43bc68df023cc WatchSource:0}: Error finding container 2493980310cb4f4afb697d4d55b351e722075b2f491eda0b88a43bc68df023cc: Status 404 returned error can't find the container with id 2493980310cb4f4afb697d4d55b351e722075b2f491eda0b88a43bc68df023cc Oct 30 13:39:34.133939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:34.133900 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:34.478853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:34.478751 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" event={"ID":"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5","Type":"ContainerStarted","Data":"eed40049badce1509ad5a031df0098011a42e2776013329bd9d56b7d239d2bf4"} Oct 30 13:39:34.478853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:34.478827 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:39:34.478853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:34.478854 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" event={"ID":"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5","Type":"ContainerStarted","Data":"2493980310cb4f4afb697d4d55b351e722075b2f491eda0b88a43bc68df023cc"} Oct 30 13:39:34.480524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:34.480489 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" event={"ID":"73d20ced-fe9a-4750-881d-835f9a94da0b","Type":"ContainerStarted","Data":"3caa1cc78b94febe5940a388c8827d52b74b7b604eb45060c5f388798c9809ff"} Oct 30 13:39:34.480659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:34.480540 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" event={"ID":"73d20ced-fe9a-4750-881d-835f9a94da0b","Type":"ContainerStarted","Data":"d6ca507c3d5f1fe1b623ce2d09f982493ef7458bd8e6b5060029218ff20b64e9"} Oct 30 13:39:34.480659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:34.480566 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:39:34.519164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:34.519109 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" podStartSLOduration=2.519097709 podStartE2EDuration="2.519097709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:34.518553002 +0000 UTC m=+2881.333747426" watchObservedRunningTime="2026-10-30 13:39:34.519097709 +0000 UTC m=+2881.334292132" Oct 30 13:39:34.561023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:34.560970 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" podStartSLOduration=2.560962125 podStartE2EDuration="2.560962125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:34.560364834 +0000 UTC m=+2881.375559257" watchObservedRunningTime="2026-10-30 13:39:34.560962125 +0000 UTC m=+2881.376156569" Oct 30 13:39:35.339970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:35.339930 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f36460-2c00-40c2-a296-9c2fd336dcd4" path="/var/lib/kubelet/pods/82f36460-2c00-40c2-a296-9c2fd336dcd4/volumes" Oct 30 13:39:35.520078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:35.514066 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:37.489588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:37.489453 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:39.636409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:39.636366 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:40.638853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:40.638816 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:40.827539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:40.827452 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:40.989721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:40.988748 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:41.440152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.440115 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69"] Oct 30 13:39:41.448482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.448453 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:41.454526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.454489 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69"] Oct 30 13:39:41.461502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.461474 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:39:41.461634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.461579 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:39:41.588050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.587990 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:41.644780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.644742 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:41.645235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.645215 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:41.645516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.645500 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lrg6\" (UniqueName: \"kubernetes.io/projected/47c36655-5532-4b45-b11a-8547a13016bf-kube-api-access-2lrg6\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:41.747119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.747022 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:41.747290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.747184 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:41.747485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.747462 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lrg6\" (UniqueName: \"kubernetes.io/projected/47c36655-5532-4b45-b11a-8547a13016bf-kube-api-access-2lrg6\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:41.747818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.747791 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:41.749775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.749752 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:41.770103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:41.770072 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lrg6\" (UniqueName: \"kubernetes.io/projected/47c36655-5532-4b45-b11a-8547a13016bf-kube-api-access-2lrg6\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:42.056080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:42.054256 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:42.068312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:42.068050 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:42.242190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:42.242152 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:42.392971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:42.391252 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:42.674916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:42.674485 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69"] Oct 30 13:39:43.056632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:43.056598 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4"] Oct 30 13:39:43.195412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:43.195273 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:43.352102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:43.352061 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:43.650166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:43.650070 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" event={"ID":"47c36655-5532-4b45-b11a-8547a13016bf","Type":"ContainerStarted","Data":"6c7b915ec80165cae01306efdcea78f7b8c1ae80472711c0a50ae21d8d7135d9"} Oct 30 13:39:43.650166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:43.650143 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" event={"ID":"47c36655-5532-4b45-b11a-8547a13016bf","Type":"ContainerStarted","Data":"5c233b3d20e3b406d3fedfd7f942e07f4d170bfad349956a69f0890cc03bb9b2"} Oct 30 13:39:43.698079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:43.697916 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" podStartSLOduration=2.697898973 podStartE2EDuration="2.697898973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:43.697701054 +0000 UTC m=+2890.512895478" watchObservedRunningTime="2026-10-30 13:39:43.697898973 +0000 UTC m=+2890.513093396" Oct 30 13:39:43.805389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:43.801798 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:44.660404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:44.660369 5049 generic.go:356] "Generic (PLEG): container finished" podID="47c36655-5532-4b45-b11a-8547a13016bf" containerID="6c7b915ec80165cae01306efdcea78f7b8c1ae80472711c0a50ae21d8d7135d9" exitCode=0 Oct 30 13:39:44.660599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:44.660429 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" event={"ID":"47c36655-5532-4b45-b11a-8547a13016bf","Type":"ContainerDied","Data":"6c7b915ec80165cae01306efdcea78f7b8c1ae80472711c0a50ae21d8d7135d9"} Oct 30 13:39:45.492443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:45.492414 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:39:45.492901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:45.492480 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:39:45.732656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:45.732621 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:45.879314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:45.879285 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:45.941276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:45.941192 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:45.987925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:45.987890 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-openshift-ca\") pod \"47c36655-5532-4b45-b11a-8547a13016bf\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " Oct 30 13:39:45.988934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:45.988903 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c36655-5532-4b45-b11a-8547a13016bf-kube-api-access-2lrg6\" (UniqueName: \"kubernetes.io/projected/47c36655-5532-4b45-b11a-8547a13016bf-kube-api-access-2lrg6\") pod \"47c36655-5532-4b45-b11a-8547a13016bf\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " Oct 30 13:39:45.989159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:45.989139 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-xks-ca\") pod \"47c36655-5532-4b45-b11a-8547a13016bf\" (UID: \"47c36655-5532-4b45-b11a-8547a13016bf\") " Oct 30 13:39:45.994629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:45.988524 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-openshift-ca" pod "47c36655-5532-4b45-b11a-8547a13016bf" (UID: "47c36655-5532-4b45-b11a-8547a13016bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:46.011574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:46.011520 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c36655-5532-4b45-b11a-8547a13016bf-kube-api-access-2lrg6" pod "47c36655-5532-4b45-b11a-8547a13016bf" (UID: "47c36655-5532-4b45-b11a-8547a13016bf"). InnerVolumeSpecName "kube-api-access-2lrg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:46.084522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:46.084466 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-xks-ca" pod "47c36655-5532-4b45-b11a-8547a13016bf" (UID: "47c36655-5532-4b45-b11a-8547a13016bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:39:46.092442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:46.092406 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lrg6\" (UniqueName: \"kubernetes.io/projected/47c36655-5532-4b45-b11a-8547a13016bf-kube-api-access-2lrg6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:46.092442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:46.092443 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:46.092692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:46.092459 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c36655-5532-4b45-b11a-8547a13016bf-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:46.706423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:46.706385 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" event={"ID":"47c36655-5532-4b45-b11a-8547a13016bf","Type":"ContainerDied","Data":"5c233b3d20e3b406d3fedfd7f942e07f4d170bfad349956a69f0890cc03bb9b2"} Oct 30 13:39:46.706923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:46.706464 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c233b3d20e3b406d3fedfd7f942e07f4d170bfad349956a69f0890cc03bb9b2" Oct 30 13:39:46.706923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:46.706517 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69" Oct 30 13:39:47.738478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:47.738432 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:48.057492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.057427 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" podUID="c1b80bf4-f870-4369-97c8-cb6a416088f3" containerName="payload-pre-processing" containerID="cri-o://d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a" gracePeriod=25 Oct 30 13:39:48.417212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.417180 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:39:48.549521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.549491 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b80bf4-f870-4369-97c8-cb6a416088f3-kube-api-access-2qzgh\" (UniqueName: \"kubernetes.io/projected/c1b80bf4-f870-4369-97c8-cb6a416088f3-kube-api-access-2qzgh\") pod \"c1b80bf4-f870-4369-97c8-cb6a416088f3\" (UID: \"c1b80bf4-f870-4369-97c8-cb6a416088f3\") " Oct 30 13:39:48.549717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.549685 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b80bf4-f870-4369-97c8-cb6a416088f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b80bf4-f870-4369-97c8-cb6a416088f3-plugins-config-volume\") pod \"c1b80bf4-f870-4369-97c8-cb6a416088f3\" (UID: \"c1b80bf4-f870-4369-97c8-cb6a416088f3\") " Oct 30 13:39:48.550054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.550029 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b80bf4-f870-4369-97c8-cb6a416088f3-plugins-config-volume" pod "c1b80bf4-f870-4369-97c8-cb6a416088f3" (UID: "c1b80bf4-f870-4369-97c8-cb6a416088f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:39:48.550376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.550357 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b80bf4-f870-4369-97c8-cb6a416088f3-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:48.551762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.551731 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b80bf4-f870-4369-97c8-cb6a416088f3-kube-api-access-2qzgh" pod "c1b80bf4-f870-4369-97c8-cb6a416088f3" (UID: "c1b80bf4-f870-4369-97c8-cb6a416088f3"). InnerVolumeSpecName "kube-api-access-2qzgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:39:48.651983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.651899 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qzgh\" (UniqueName: \"kubernetes.io/projected/c1b80bf4-f870-4369-97c8-cb6a416088f3-kube-api-access-2qzgh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:39:48.730363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.730310 5049 generic.go:356] "Generic (PLEG): container finished" podID="c1b80bf4-f870-4369-97c8-cb6a416088f3" containerID="d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a" exitCode=0 Oct 30 13:39:48.730541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.730372 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" event={"ID":"c1b80bf4-f870-4369-97c8-cb6a416088f3","Type":"ContainerDied","Data":"d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a"} Oct 30 13:39:48.730541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.730398 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" Oct 30 13:39:48.730541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.730434 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4" event={"ID":"c1b80bf4-f870-4369-97c8-cb6a416088f3","Type":"ContainerDied","Data":"f064f643c7fc215ce6f5870c416bb09e1e3941862393eff4a6595ea3036f2277"} Oct 30 13:39:48.730541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.730461 5049 scope.go:122] "RemoveContainer" containerID="d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a" Oct 30 13:39:48.745374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.745352 5049 scope.go:122] "RemoveContainer" containerID="d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a" Oct 30 13:39:48.745997 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:39:48.745977 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a\": container with ID starting with d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a not found: ID does not exist" containerID="d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a" Oct 30 13:39:48.746070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.746004 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a"} err="failed to get container status \"d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a\": rpc error: code = NotFound desc = could not find container \"d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a\": container with ID starting with d91151f41f66da91765795f6762f00ab4f7f7de4b4c0aaf8ef493a695d5e1e1a not found: ID does not exist" Oct 30 13:39:48.781426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.781393 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4"] Oct 30 13:39:48.792495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:48.792462 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-7476c67cd7-482s4"] Oct 30 13:39:49.340169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:49.340127 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b80bf4-f870-4369-97c8-cb6a416088f3" path="/var/lib/kubelet/pods/c1b80bf4-f870-4369-97c8-cb6a416088f3/volumes" Oct 30 13:39:49.741259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:49.741163 5049 generic.go:356] "Generic (PLEG): container finished" podID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerID="2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8" exitCode=0 Oct 30 13:39:49.741259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:49.741236 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" event={"ID":"574a0063-5e8f-448f-b5d2-a2f182fdbc25","Type":"ContainerDied","Data":"2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8"} Oct 30 13:39:49.741798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:49.741773 5049 scope.go:122] "RemoveContainer" containerID="2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8" Oct 30 13:39:49.744513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:49.744476 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:49.744613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:49.744564 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:49.744663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:49.744616 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:39:50.155134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:50.153394 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:50.754688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:50.754645 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" event={"ID":"574a0063-5e8f-448f-b5d2-a2f182fdbc25","Type":"ContainerStarted","Data":"40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e"} Oct 30 13:39:50.938980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:50.938939 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:39:50.939278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:50.939099 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 13:39:50.939501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:50.939477 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 13:39:51.877779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:51.877741 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n"] Oct 30 13:39:51.940100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:51.939853 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 13:39:51.940100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:51.939914 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 13:39:51.940100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:51.940048 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh"] Oct 30 13:39:51.942883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:51.942851 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:52.132572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:52.115416 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:52.941700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:52.941382 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 13:39:52.941700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:52.941451 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 13:39:53.901344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:53.899436 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:53.942211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:53.941580 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 13:39:53.942211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:53.941665 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 13:39:54.942344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:54.939802 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 13:39:54.942344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:54.939862 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 13:39:55.726344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:55.724798 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:55.939147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:55.939102 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 30 13:39:55.939387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:55.939175 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 30 13:39:56.684702 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:39:56.684666 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:39:56.878137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:56.878097 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" podUID="9a90fb13-284a-4c86-b7b9-2598d28a392e" containerName="payload-processing" containerID="cri-o://bf117999f27b80603f58c7e43cfbde0baf4b878b621a7bc2e64dbef8394a1285" gracePeriod=25 Oct 30 13:39:56.941441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:56.941348 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" podUID="45b1cb66-7012-4411-ad37-b8fe9e24b586" containerName="payload-pre-processing" containerID="cri-o://e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e" gracePeriod=25 Oct 30 13:39:56.945494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:56.945467 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:39:56.946009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:56.945987 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:39:56.946820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:56.946798 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:39:58.104034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:39:58.103995 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:00.016346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:00.014948 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:01.806642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:01.806596 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:02.708287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:02.708242 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" podUID="45b1cb66-7012-4411-ad37-b8fe9e24b586" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:40:02.708541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:02.708442 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" podUID="9a90fb13-284a-4c86-b7b9-2598d28a392e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:40:03.620260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:03.620207 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:05.540911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:05.540857 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:05.704466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:05.704431 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:05.876666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:05.876617 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:07.518770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:07.518734 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:09.964159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:09.949513 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:12.544726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:12.544641 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:12.708406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:12.708368 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" podUID="9a90fb13-284a-4c86-b7b9-2598d28a392e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:40:12.708617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:12.708370 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" podUID="45b1cb66-7012-4411-ad37-b8fe9e24b586" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:40:14.984093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:14.984013 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:16.812724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:16.812687 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:17.057724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.057692 5049 generic.go:356] "Generic (PLEG): container finished" podID="9a90fb13-284a-4c86-b7b9-2598d28a392e" containerID="bf117999f27b80603f58c7e43cfbde0baf4b878b621a7bc2e64dbef8394a1285" exitCode=0 Oct 30 13:40:17.057855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.057768 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" event={"ID":"9a90fb13-284a-4c86-b7b9-2598d28a392e","Type":"ContainerDied","Data":"bf117999f27b80603f58c7e43cfbde0baf4b878b621a7bc2e64dbef8394a1285"} Oct 30 13:40:17.265878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.265845 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:40:17.305288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.305264 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:40:17.412657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.412562 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45b1cb66-7012-4411-ad37-b8fe9e24b586-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b1cb66-7012-4411-ad37-b8fe9e24b586-plugins-config-volume\") pod \"45b1cb66-7012-4411-ad37-b8fe9e24b586\" (UID: \"45b1cb66-7012-4411-ad37-b8fe9e24b586\") " Oct 30 13:40:17.412922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.412888 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a90fb13-284a-4c86-b7b9-2598d28a392e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a90fb13-284a-4c86-b7b9-2598d28a392e-plugins-config-volume\") pod \"9a90fb13-284a-4c86-b7b9-2598d28a392e\" (UID: \"9a90fb13-284a-4c86-b7b9-2598d28a392e\") " Oct 30 13:40:17.413047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.413024 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a90fb13-284a-4c86-b7b9-2598d28a392e-kube-api-access-947db\" (UniqueName: \"kubernetes.io/projected/9a90fb13-284a-4c86-b7b9-2598d28a392e-kube-api-access-947db\") pod \"9a90fb13-284a-4c86-b7b9-2598d28a392e\" (UID: \"9a90fb13-284a-4c86-b7b9-2598d28a392e\") " Oct 30 13:40:17.413249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.413231 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b1cb66-7012-4411-ad37-b8fe9e24b586-kube-api-access-h4kz4\" (UniqueName: \"kubernetes.io/projected/45b1cb66-7012-4411-ad37-b8fe9e24b586-kube-api-access-h4kz4\") pod \"45b1cb66-7012-4411-ad37-b8fe9e24b586\" (UID: \"45b1cb66-7012-4411-ad37-b8fe9e24b586\") " Oct 30 13:40:17.416200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.416172 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a90fb13-284a-4c86-b7b9-2598d28a392e-plugins-config-volume" pod "9a90fb13-284a-4c86-b7b9-2598d28a392e" (UID: "9a90fb13-284a-4c86-b7b9-2598d28a392e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:17.416614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.416585 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b1cb66-7012-4411-ad37-b8fe9e24b586-plugins-config-volume" pod "45b1cb66-7012-4411-ad37-b8fe9e24b586" (UID: "45b1cb66-7012-4411-ad37-b8fe9e24b586"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:17.417558 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.417533 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b1cb66-7012-4411-ad37-b8fe9e24b586-kube-api-access-h4kz4" pod "45b1cb66-7012-4411-ad37-b8fe9e24b586" (UID: "45b1cb66-7012-4411-ad37-b8fe9e24b586"). InnerVolumeSpecName "kube-api-access-h4kz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:17.418084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.418064 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a90fb13-284a-4c86-b7b9-2598d28a392e-kube-api-access-947db" pod "9a90fb13-284a-4c86-b7b9-2598d28a392e" (UID: "9a90fb13-284a-4c86-b7b9-2598d28a392e"). InnerVolumeSpecName "kube-api-access-947db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:17.515347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.515300 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4kz4\" (UniqueName: \"kubernetes.io/projected/45b1cb66-7012-4411-ad37-b8fe9e24b586-kube-api-access-h4kz4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:17.515347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.515344 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b1cb66-7012-4411-ad37-b8fe9e24b586-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:17.515347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.515354 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a90fb13-284a-4c86-b7b9-2598d28a392e-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:17.515609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:17.515364 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-947db\" (UniqueName: \"kubernetes.io/projected/9a90fb13-284a-4c86-b7b9-2598d28a392e-kube-api-access-947db\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:18.069836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.069808 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" Oct 30 13:40:18.070356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.069850 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n" event={"ID":"9a90fb13-284a-4c86-b7b9-2598d28a392e","Type":"ContainerDied","Data":"a73db91e415fa71ae4cfece9009991e39b05aa97d52f8ecb56b56824eb9b5438"} Oct 30 13:40:18.070356 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.069920 5049 scope.go:122] "RemoveContainer" containerID="bf117999f27b80603f58c7e43cfbde0baf4b878b621a7bc2e64dbef8394a1285" Oct 30 13:40:18.071644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.071612 5049 generic.go:356] "Generic (PLEG): container finished" podID="45b1cb66-7012-4411-ad37-b8fe9e24b586" containerID="e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e" exitCode=0 Oct 30 13:40:18.071747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.071673 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" Oct 30 13:40:18.071747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.071697 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" event={"ID":"45b1cb66-7012-4411-ad37-b8fe9e24b586","Type":"ContainerDied","Data":"e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e"} Oct 30 13:40:18.071840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.071752 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh" event={"ID":"45b1cb66-7012-4411-ad37-b8fe9e24b586","Type":"ContainerDied","Data":"6b5c7e40a3feb57678355dd5ef10550265093b41106c10b89b06fa74c24aae62"} Oct 30 13:40:18.085266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.085248 5049 scope.go:122] "RemoveContainer" containerID="e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e" Oct 30 13:40:18.096222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.096194 5049 scope.go:122] "RemoveContainer" containerID="e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e" Oct 30 13:40:18.096584 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:40:18.096564 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e\": container with ID starting with e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e not found: ID does not exist" containerID="e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e" Oct 30 13:40:18.096663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.096590 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e"} err="failed to get container status \"e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e\": rpc error: code = NotFound desc = could not find container \"e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e\": container with ID starting with e592819edb4d4f113b39d8985179400f00b61a4eef077d46e1bcd8131440f39e not found: ID does not exist" Oct 30 13:40:18.132820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.132785 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n"] Oct 30 13:40:18.148976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.148889 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5a6281-684c8bf98d-nnv6n"] Oct 30 13:40:18.168457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.168419 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh"] Oct 30 13:40:18.179112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.179077 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5a6281-84dcdb8795-j66rh"] Oct 30 13:40:18.810937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:18.810035 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:19.340194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:19.340151 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b1cb66-7012-4411-ad37-b8fe9e24b586" path="/var/lib/kubelet/pods/45b1cb66-7012-4411-ad37-b8fe9e24b586/volumes" Oct 30 13:40:19.340891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:19.340867 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a90fb13-284a-4c86-b7b9-2598d28a392e" path="/var/lib/kubelet/pods/9a90fb13-284a-4c86-b7b9-2598d28a392e/volumes" Oct 30 13:40:20.541254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:20.541216 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:22.294437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:22.294286 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:24.378797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:24.378758 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:25.201878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.201788 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:25.367139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.367102 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:25.521981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.521883 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:25.827712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.827673 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b"] Oct 30 13:40:25.829071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829040 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9a90fb13-284a-4c86-b7b9-2598d28a392e" containerName="payload-processing" Oct 30 13:40:25.829071 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829071 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a90fb13-284a-4c86-b7b9-2598d28a392e" containerName="payload-processing" Oct 30 13:40:25.829291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829120 5049 cpu_manager.go:414] "containerMap: removing container" podUID="45b1cb66-7012-4411-ad37-b8fe9e24b586" containerName="payload-pre-processing" Oct 30 13:40:25.829291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829130 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b1cb66-7012-4411-ad37-b8fe9e24b586" containerName="payload-pre-processing" Oct 30 13:40:25.829291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829154 5049 cpu_manager.go:414] "containerMap: removing container" podUID="47c36655-5532-4b45-b11a-8547a13016bf" containerName="revoke-keys" Oct 30 13:40:25.829291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829162 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c36655-5532-4b45-b11a-8547a13016bf" containerName="revoke-keys" Oct 30 13:40:25.829291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829180 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c1b80bf4-f870-4369-97c8-cb6a416088f3" containerName="payload-pre-processing" Oct 30 13:40:25.829291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829190 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b80bf4-f870-4369-97c8-cb6a416088f3" containerName="payload-pre-processing" Oct 30 13:40:25.829633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829501 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a90fb13-284a-4c86-b7b9-2598d28a392e" containerName="payload-processing" Oct 30 13:40:25.829633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829525 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b1cb66-7012-4411-ad37-b8fe9e24b586" containerName="payload-pre-processing" Oct 30 13:40:25.829633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829538 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c36655-5532-4b45-b11a-8547a13016bf" containerName="revoke-keys" Oct 30 13:40:25.829633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.829549 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b80bf4-f870-4369-97c8-cb6a416088f3" containerName="payload-pre-processing" Oct 30 13:40:25.836334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.836293 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.852001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.849513 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b"] Oct 30 13:40:25.854959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.854930 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b4fb25-openshift-default-dockercfg-5469m\"" Oct 30 13:40:25.881029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.880996 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805a405c-36db-43a0-b270-d33398a5d952-istiod-ca-cert\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.881263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.881240 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-envoy\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.881391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.881276 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-istio-token\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.881487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.881465 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-credential-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.881564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.881537 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805a405c-36db-43a0-b270-d33398a5d952-istio-podinfo\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.881626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.881598 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-data\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.881915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.881843 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-certs\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.881915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.881898 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.882077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.881956 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp2sc\" (UniqueName: \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-kube-api-access-dp2sc\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.983725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.983690 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-credential-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.983937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.983748 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805a405c-36db-43a0-b270-d33398a5d952-istio-podinfo\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.983937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.983793 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-data\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.983937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.983832 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-certs\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.983937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.983876 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.983937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.983927 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp2sc\" (UniqueName: \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-kube-api-access-dp2sc\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.984229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.984158 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-credential-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.984291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.984228 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-certs\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.984291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.984275 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.984497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.984471 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805a405c-36db-43a0-b270-d33398a5d952-istiod-ca-cert\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.984560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.984483 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-data\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.984697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.984674 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-envoy\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.984851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.984714 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-istio-token\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.985201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.985171 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805a405c-36db-43a0-b270-d33398a5d952-istiod-ca-cert\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.987102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.987076 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805a405c-36db-43a0-b270-d33398a5d952-istio-podinfo\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:25.987251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:25.987229 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-envoy\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:26.006366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:26.006334 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-istio-token\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:26.006973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:26.006948 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp2sc\" (UniqueName: \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-kube-api-access-dp2sc\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:26.160068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:26.159961 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:26.515724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:26.515615 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:26.551597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:26.551569 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b"] Oct 30 13:40:26.553947 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:40:26.553920 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805a405c_36db_43a0_b270_d33398a5d952.slice/crio-f35fbde40272883428eb7369a9b3f93ceb5904aed6613178fb8eb505c0c5a165 WatchSource:0}: Error finding container f35fbde40272883428eb7369a9b3f93ceb5904aed6613178fb8eb505c0c5a165: Status 404 returned error can't find the container with id f35fbde40272883428eb7369a9b3f93ceb5904aed6613178fb8eb505c0c5a165 Oct 30 13:40:26.557500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:26.557467 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:26.557617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:26.557545 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:26.557617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:26.557591 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:26.703887 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:40:26.703853 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:40:26.748862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:26.748780 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:27.203650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:27.203608 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" event={"ID":"805a405c-36db-43a0-b270-d33398a5d952","Type":"ContainerStarted","Data":"4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8"} Oct 30 13:40:27.203901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:27.203670 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" event={"ID":"805a405c-36db-43a0-b270-d33398a5d952","Type":"ContainerStarted","Data":"f35fbde40272883428eb7369a9b3f93ceb5904aed6613178fb8eb505c0c5a165"} Oct 30 13:40:27.265869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:27.265816 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" podStartSLOduration=2.265804274 podStartE2EDuration="2.265804274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:27.264661909 +0000 UTC m=+2934.079856331" watchObservedRunningTime="2026-10-30 13:40:27.265804274 +0000 UTC m=+2934.080998697" Oct 30 13:40:28.160616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:28.160582 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:28.162459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:28.162430 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 13:40:28.162567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:28.162506 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" podUID="805a405c-36db-43a0-b270-d33398a5d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 13:40:28.577699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:28.571376 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:29.040970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:29.040893 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx"] Oct 30 13:40:29.047754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:29.047724 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1ba611-lhppx"] Oct 30 13:40:29.161079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:29.161048 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 13:40:29.161484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:29.161104 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" podUID="805a405c-36db-43a0-b270-d33398a5d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 13:40:29.344386 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:29.344350 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7" path="/var/lib/kubelet/pods/54f6d1e6-5f55-4ce1-9e5b-ac319ebc16d7/volumes" Oct 30 13:40:29.975809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:29.975767 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:30.160642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:30.160604 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 13:40:30.160827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:30.160671 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" podUID="805a405c-36db-43a0-b270-d33398a5d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 13:40:30.580876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:30.580819 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:31.160623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:31.160579 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 13:40:31.160786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:31.160657 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" podUID="805a405c-36db-43a0-b270-d33398a5d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 13:40:32.161033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:32.160995 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 13:40:32.161683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:32.161638 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" podUID="805a405c-36db-43a0-b270-d33398a5d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 13:40:32.561393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:32.561356 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:33.246624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:33.246578 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:33.247397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:33.247363 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:33.248558 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:33.248525 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:40:33.850010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:33.849980 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:34.020908 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:34.020871 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:35.034810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:35.034769 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg"] Oct 30 13:40:35.045221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:35.045184 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33aa610-q9jvg"] Oct 30 13:40:35.179288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:35.179191 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:35.339147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:35.339105 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ce81e3-250c-4960-ad10-9efa2f145e60" path="/var/lib/kubelet/pods/62ce81e3-250c-4960-ad10-9efa2f145e60/volumes" Oct 30 13:40:36.529300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:36.529264 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:36.754770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:36.754736 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:37.303298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:37.303258 5049 scope.go:122] "RemoveContainer" containerID="00ced1778ea658c177bac557254551617094586f3a41a0c8457a32dd41c09c7b" Oct 30 13:40:37.326101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:37.326076 5049 scope.go:122] "RemoveContainer" containerID="f67687d9a123a0f33ef133567718b77034d294e46c36ede672456d159857d3e2" Oct 30 13:40:37.339364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:37.339336 5049 scope.go:122] "RemoveContainer" containerID="6e2bdb8d45f09af275e2039a0d109c8f8c2466937c7ea5b1b61fab53a5d66da9" Oct 30 13:40:37.350380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:37.350357 5049 scope.go:122] "RemoveContainer" containerID="f67693126adce99073d49c1aa40351d83f3db7802185d16cdc0f14d14d9f4934" Oct 30 13:40:37.361267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:37.361245 5049 scope.go:122] "RemoveContainer" containerID="8d5e9a5e42162bb9c329ac428faf6e742fd8e051f0d2f9839be9f623afe5270a" Oct 30 13:40:37.374564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:37.374437 5049 scope.go:122] "RemoveContainer" containerID="d876b2e4c2f540f8892d998eb00f17cc349a11182383dbbb5eb1bb9fc5993005" Oct 30 13:40:37.828228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:37.828075 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:39.708209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:39.708159 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:40.033267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:40.033169 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9"] Oct 30 13:40:40.042578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:40.042541 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef9da70-g22n9"] Oct 30 13:40:41.343457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:41.343420 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8aaedb-2406-4287-9faa-6a7f261b3cc5" path="/var/lib/kubelet/pods/0a8aaedb-2406-4287-9faa-6a7f261b3cc5/volumes" Oct 30 13:40:41.344063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:41.344041 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g"] Oct 30 13:40:41.344306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:41.344278 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerName="istio-proxy" containerID="cri-o://9ec0c406c43ea80acbc725bb189e6d2ff05182b6517ee5304975a94dceb9d477" gracePeriod=30 Oct 30 13:40:41.501146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:41.498555 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:43.340983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:43.340893 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:45.033514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:45.033469 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l"] Oct 30 13:40:45.042565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:45.042533 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3bb89b-hsh7l"] Oct 30 13:40:45.325409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:45.325374 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:45.363341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:45.354406 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8ae58c-a49e-46ec-9231-21e6fa097adb" path="/var/lib/kubelet/pods/ef8ae58c-a49e-46ec-9231-21e6fa097adb/volumes" Oct 30 13:40:46.650854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.650776 5049 generic.go:356] "Generic (PLEG): container finished" podID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerID="9ec0c406c43ea80acbc725bb189e6d2ff05182b6517ee5304975a94dceb9d477" exitCode=0 Oct 30 13:40:46.651388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.650953 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" event={"ID":"ee84f7f1-7d54-4354-bd2b-872e0cb59b75","Type":"ContainerDied","Data":"9ec0c406c43ea80acbc725bb189e6d2ff05182b6517ee5304975a94dceb9d477"} Oct 30 13:40:46.742277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.742245 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:40:46.746195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746173 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istiod-ca-cert\") pod \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " Oct 30 13:40:46.746297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746263 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-data\") pod \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " Oct 30 13:40:46.746297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746285 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-podinfo\") pod \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " Oct 30 13:40:46.746390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746376 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-certs\") pod \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " Oct 30 13:40:46.746555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746521 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-token\" (UniqueName: \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-token\") pod \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " Oct 30 13:40:46.746555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746546 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istiod-ca-cert" pod "ee84f7f1-7d54-4354-bd2b-872e0cb59b75" (UID: "ee84f7f1-7d54-4354-bd2b-872e0cb59b75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:46.746751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746566 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-socket\") pod \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " Oct 30 13:40:46.746751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746597 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-data" pod "ee84f7f1-7d54-4354-bd2b-872e0cb59b75" (UID: "ee84f7f1-7d54-4354-bd2b-872e0cb59b75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:46.746751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746668 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-credential-socket\") pod \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " Oct 30 13:40:46.746751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746704 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-certs" pod "ee84f7f1-7d54-4354-bd2b-872e0cb59b75" (UID: "ee84f7f1-7d54-4354-bd2b-872e0cb59b75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:46.746751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746726 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-socket" pod "ee84f7f1-7d54-4354-bd2b-872e0cb59b75" (UID: "ee84f7f1-7d54-4354-bd2b-872e0cb59b75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:46.746751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746737 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-kube-api-access-c6v74\" (UniqueName: \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-kube-api-access-c6v74\") pod \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " Oct 30 13:40:46.747122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746819 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-envoy\") pod \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\" (UID: \"ee84f7f1-7d54-4354-bd2b-872e0cb59b75\") " Oct 30 13:40:46.747122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.746932 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-credential-socket" pod "ee84f7f1-7d54-4354-bd2b-872e0cb59b75" (UID: "ee84f7f1-7d54-4354-bd2b-872e0cb59b75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:46.747652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.747629 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.747750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.747656 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.747750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.747669 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.747750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.747682 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.747750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.747694 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.748996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.748969 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-token" pod "ee84f7f1-7d54-4354-bd2b-872e0cb59b75" (UID: "ee84f7f1-7d54-4354-bd2b-872e0cb59b75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:46.749139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.749121 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-podinfo" pod "ee84f7f1-7d54-4354-bd2b-872e0cb59b75" (UID: "ee84f7f1-7d54-4354-bd2b-872e0cb59b75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:40:46.749247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.749229 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-kube-api-access-c6v74" pod "ee84f7f1-7d54-4354-bd2b-872e0cb59b75" (UID: "ee84f7f1-7d54-4354-bd2b-872e0cb59b75"). InnerVolumeSpecName "kube-api-access-c6v74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:46.749472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.749457 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-envoy" pod "ee84f7f1-7d54-4354-bd2b-872e0cb59b75" (UID: "ee84f7f1-7d54-4354-bd2b-872e0cb59b75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:46.849581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.849544 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.849581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.849575 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6v74\" (UniqueName: \"kubernetes.io/projected/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-kube-api-access-c6v74\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.849581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.849586 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:46.849827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:46.849595 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee84f7f1-7d54-4354-bd2b-872e0cb59b75-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:47.630497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:47.630384 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp"] Oct 30 13:40:47.630969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:47.630937 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" podUID="5954f9fb-eadc-458f-a39d-678fc56e6e5e" containerName="maas-api" containerID="cri-o://5fdf11629156c8e8f3b1b5019663fae72cce27814ed037ad870ac3076e990b5d" gracePeriod=30 Oct 30 13:40:47.686861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:47.686827 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:47.691087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:47.691054 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" event={"ID":"ee84f7f1-7d54-4354-bd2b-872e0cb59b75","Type":"ContainerDied","Data":"79ea4d9b245655a1e2dd9236cebd1d93f274cb52b3c57d7000d6bcbf807dc1ec"} Oct 30 13:40:47.691221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:47.691128 5049 scope.go:122] "RemoveContainer" containerID="9ec0c406c43ea80acbc725bb189e6d2ff05182b6517ee5304975a94dceb9d477" Oct 30 13:40:47.691304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:47.691288 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g" Oct 30 13:40:47.977400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:47.975232 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g"] Oct 30 13:40:48.006247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:47.994385 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-xdn6g"] Oct 30 13:40:48.579457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.576139 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:48.588774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.588741 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69"] Oct 30 13:40:48.595951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.595919 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5a6281-mdj69"] Oct 30 13:40:48.716547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.716453 5049 generic.go:356] "Generic (PLEG): container finished" podID="5954f9fb-eadc-458f-a39d-678fc56e6e5e" containerID="5fdf11629156c8e8f3b1b5019663fae72cce27814ed037ad870ac3076e990b5d" exitCode=0 Oct 30 13:40:48.717142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.717117 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" event={"ID":"5954f9fb-eadc-458f-a39d-678fc56e6e5e","Type":"ContainerDied","Data":"5fdf11629156c8e8f3b1b5019663fae72cce27814ed037ad870ac3076e990b5d"} Oct 30 13:40:48.743783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.743749 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:48.865567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.865527 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:40:48.989504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.989406 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-metrics-certs\") pod \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " Oct 30 13:40:48.989685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.989515 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5954f9fb-eadc-458f-a39d-678fc56e6e5e-kube-api-access-j5gr8\" (UniqueName: \"kubernetes.io/projected/5954f9fb-eadc-458f-a39d-678fc56e6e5e-kube-api-access-j5gr8\") pod \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " Oct 30 13:40:48.989801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.989784 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5954f9fb-eadc-458f-a39d-678fc56e6e5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5954f9fb-eadc-458f-a39d-678fc56e6e5e-openshift-service-ca\") pod \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " Oct 30 13:40:48.989890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.989830 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-maas-api-tls\") pod \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\" (UID: \"5954f9fb-eadc-458f-a39d-678fc56e6e5e\") " Oct 30 13:40:48.990850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.990817 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5954f9fb-eadc-458f-a39d-678fc56e6e5e-openshift-service-ca" pod "5954f9fb-eadc-458f-a39d-678fc56e6e5e" (UID: "5954f9fb-eadc-458f-a39d-678fc56e6e5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:48.993538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.993508 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-metrics-certs" pod "5954f9fb-eadc-458f-a39d-678fc56e6e5e" (UID: "5954f9fb-eadc-458f-a39d-678fc56e6e5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:48.993686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.993523 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-maas-api-tls" pod "5954f9fb-eadc-458f-a39d-678fc56e6e5e" (UID: "5954f9fb-eadc-458f-a39d-678fc56e6e5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:48.995868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:48.995806 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5954f9fb-eadc-458f-a39d-678fc56e6e5e-kube-api-access-j5gr8" pod "5954f9fb-eadc-458f-a39d-678fc56e6e5e" (UID: "5954f9fb-eadc-458f-a39d-678fc56e6e5e"). InnerVolumeSpecName "kube-api-access-j5gr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:49.092593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.092554 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5954f9fb-eadc-458f-a39d-678fc56e6e5e-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:49.092593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.092596 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:49.092821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.092611 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5954f9fb-eadc-458f-a39d-678fc56e6e5e-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:49.092821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.092624 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5gr8\" (UniqueName: \"kubernetes.io/projected/5954f9fb-eadc-458f-a39d-678fc56e6e5e-kube-api-access-j5gr8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:49.094504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.094478 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:49.374506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.374462 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c36655-5532-4b45-b11a-8547a13016bf" path="/var/lib/kubelet/pods/47c36655-5532-4b45-b11a-8547a13016bf/volumes" Oct 30 13:40:49.376462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.376360 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" path="/var/lib/kubelet/pods/ee84f7f1-7d54-4354-bd2b-872e0cb59b75/volumes" Oct 30 13:40:49.763210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.763087 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" event={"ID":"5954f9fb-eadc-458f-a39d-678fc56e6e5e","Type":"ContainerDied","Data":"ab09195ef53f3fb2ff1ad2c83e4ea50f37cd66afcd4887c41000542605cad62c"} Oct 30 13:40:49.763210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.763193 5049 scope.go:122] "RemoveContainer" containerID="5fdf11629156c8e8f3b1b5019663fae72cce27814ed037ad870ac3076e990b5d" Oct 30 13:40:49.764343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.763375 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp" Oct 30 13:40:49.862107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.855053 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp"] Oct 30 13:40:49.862107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:49.862002 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5a6281-6bbb86d67-pbbtp"] Oct 30 13:40:50.520348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:50.519982 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:51.072289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.062400 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r"] Oct 30 13:40:51.072289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.063762 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5954f9fb-eadc-458f-a39d-678fc56e6e5e" containerName="maas-api" Oct 30 13:40:51.072289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.063783 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5954f9fb-eadc-458f-a39d-678fc56e6e5e" containerName="maas-api" Oct 30 13:40:51.072289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.063831 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerName="istio-proxy" Oct 30 13:40:51.072289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.063839 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerName="istio-proxy" Oct 30 13:40:51.072289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.064073 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5954f9fb-eadc-458f-a39d-678fc56e6e5e" containerName="maas-api" Oct 30 13:40:51.072289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.064095 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee84f7f1-7d54-4354-bd2b-872e0cb59b75" containerName="istio-proxy" Oct 30 13:40:51.074042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.073278 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.085464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.085439 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r"] Oct 30 13:40:51.088058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.088035 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:40:51.088190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.088169 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:40:51.174637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.174510 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dsf5\" (UniqueName: \"kubernetes.io/projected/72d542fa-4886-4c68-a55f-19c1fbee1837-kube-api-access-5dsf5\") pod \"maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.178518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.175203 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.178518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.175270 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.278108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.278068 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dsf5\" (UniqueName: \"kubernetes.io/projected/72d542fa-4886-4c68-a55f-19c1fbee1837-kube-api-access-5dsf5\") pod \"maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.278292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.278135 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.278292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.278176 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.278914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.278887 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.281007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.280988 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.301123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.301092 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dsf5\" (UniqueName: \"kubernetes.io/projected/72d542fa-4886-4c68-a55f-19c1fbee1837-kube-api-access-5dsf5\") pod \"maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.375456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.375375 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5954f9fb-eadc-458f-a39d-678fc56e6e5e" path="/var/lib/kubelet/pods/5954f9fb-eadc-458f-a39d-678fc56e6e5e/volumes" Oct 30 13:40:51.394151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.393907 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:51.948517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:51.948483 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r"] Oct 30 13:40:51.949192 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:40:51.949163 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d542fa_4886_4c68_a55f_19c1fbee1837.slice/crio-92dbcc76bc6745e728612a5901ad3789529c48605c20fcfb8a06e2da9b497576 WatchSource:0}: Error finding container 92dbcc76bc6745e728612a5901ad3789529c48605c20fcfb8a06e2da9b497576: Status 404 returned error can't find the container with id 92dbcc76bc6745e728612a5901ad3789529c48605c20fcfb8a06e2da9b497576 Oct 30 13:40:52.534399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:52.530271 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:52.798911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:52.798866 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" event={"ID":"72d542fa-4886-4c68-a55f-19c1fbee1837","Type":"ContainerStarted","Data":"8fa778d7720d7f1b390f39a08c2127c097a9b14a3e15b6aeaf078d070145ea63"} Oct 30 13:40:52.799095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:52.798934 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" event={"ID":"72d542fa-4886-4c68-a55f-19c1fbee1837","Type":"ContainerStarted","Data":"92dbcc76bc6745e728612a5901ad3789529c48605c20fcfb8a06e2da9b497576"} Oct 30 13:40:52.856830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:52.856762 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" podStartSLOduration=1.856744602 podStartE2EDuration="1.856744602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:52.854353143 +0000 UTC m=+2959.669547560" watchObservedRunningTime="2026-10-30 13:40:52.856744602 +0000 UTC m=+2959.671939026" Oct 30 13:40:53.813944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:53.813908 5049 generic.go:356] "Generic (PLEG): container finished" podID="72d542fa-4886-4c68-a55f-19c1fbee1837" containerID="8fa778d7720d7f1b390f39a08c2127c097a9b14a3e15b6aeaf078d070145ea63" exitCode=0 Oct 30 13:40:53.814438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:53.813953 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" event={"ID":"72d542fa-4886-4c68-a55f-19c1fbee1837","Type":"ContainerDied","Data":"8fa778d7720d7f1b390f39a08c2127c097a9b14a3e15b6aeaf078d070145ea63"} Oct 30 13:40:54.896130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:54.895134 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:55.146832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.146734 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:55.180700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.180659 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-xks-ca\") pod \"72d542fa-4886-4c68-a55f-19c1fbee1837\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " Oct 30 13:40:55.180897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.180725 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-openshift-ca\") pod \"72d542fa-4886-4c68-a55f-19c1fbee1837\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " Oct 30 13:40:55.180897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.180835 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d542fa-4886-4c68-a55f-19c1fbee1837-kube-api-access-5dsf5\" (UniqueName: \"kubernetes.io/projected/72d542fa-4886-4c68-a55f-19c1fbee1837-kube-api-access-5dsf5\") pod \"72d542fa-4886-4c68-a55f-19c1fbee1837\" (UID: \"72d542fa-4886-4c68-a55f-19c1fbee1837\") " Oct 30 13:40:55.182861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.182831 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-xks-ca" pod "72d542fa-4886-4c68-a55f-19c1fbee1837" (UID: "72d542fa-4886-4c68-a55f-19c1fbee1837"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:40:55.185953 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.185925 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d542fa-4886-4c68-a55f-19c1fbee1837-kube-api-access-5dsf5" pod "72d542fa-4886-4c68-a55f-19c1fbee1837" (UID: "72d542fa-4886-4c68-a55f-19c1fbee1837"). InnerVolumeSpecName "kube-api-access-5dsf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:55.221385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.218999 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-openshift-ca" pod "72d542fa-4886-4c68-a55f-19c1fbee1837" (UID: "72d542fa-4886-4c68-a55f-19c1fbee1837"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:55.283502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.283469 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dsf5\" (UniqueName: \"kubernetes.io/projected/72d542fa-4886-4c68-a55f-19c1fbee1837-kube-api-access-5dsf5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:55.283502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.283499 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:55.283502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.283509 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d542fa-4886-4c68-a55f-19c1fbee1837-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:40:55.668493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.668452 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:55.832049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.832016 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:55.850174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.850140 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" Oct 30 13:40:55.850174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.850154 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r" event={"ID":"72d542fa-4886-4c68-a55f-19c1fbee1837","Type":"ContainerDied","Data":"92dbcc76bc6745e728612a5901ad3789529c48605c20fcfb8a06e2da9b497576"} Oct 30 13:40:55.850438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.850207 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92dbcc76bc6745e728612a5901ad3789529c48605c20fcfb8a06e2da9b497576" Oct 30 13:40:55.925796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.923442 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:40:55.925796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.923582 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:40:55.936340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.935782 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b"] Oct 30 13:40:55.936340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.936021 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" containerID="cri-o://10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c" gracePeriod=30 Oct 30 13:40:55.988567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:55.988528 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:56.486884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.486848 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:56.680517 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:40:56.680472 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:40:56.683382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.683352 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt"] Oct 30 13:40:56.684239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.684218 5049 cpu_manager.go:414] "containerMap: removing container" podUID="72d542fa-4886-4c68-a55f-19c1fbee1837" containerName="revoke-keys" Oct 30 13:40:56.684239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.684236 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d542fa-4886-4c68-a55f-19c1fbee1837" containerName="revoke-keys" Oct 30 13:40:56.684443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.684420 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d542fa-4886-4c68-a55f-19c1fbee1837" containerName="revoke-keys" Oct 30 13:40:56.732615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.732578 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt"] Oct 30 13:40:56.732796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.732696 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:56.748287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.748259 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:40:56.758549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.758507 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:40:56.855851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.850273 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:56.856151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.856128 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqrb\" (UniqueName: \"kubernetes.io/projected/5287534f-89cd-4048-87a5-4d3039a9ac7d-kube-api-access-clqrb\") pod \"maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:56.856233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.856217 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:56.974292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.974245 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:56.974907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.974695 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqrb\" (UniqueName: \"kubernetes.io/projected/5287534f-89cd-4048-87a5-4d3039a9ac7d-kube-api-access-clqrb\") pod \"maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:56.974907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.974830 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:56.975924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.975647 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:56.976727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.976649 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:56.979222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:56.979200 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:57.007389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:57.006792 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqrb\" (UniqueName: \"kubernetes.io/projected/5287534f-89cd-4048-87a5-4d3039a9ac7d-kube-api-access-clqrb\") pod \"maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:57.113882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:57.113843 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:40:57.511867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:57.511837 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt"] Oct 30 13:40:57.513350 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:40:57.513291 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5287534f_89cd_4048_87a5_4d3039a9ac7d.slice/crio-92517de304b0faf30ddc0ba0b9b1de046caf061136510b8f3e22b947c0a838f2 WatchSource:0}: Error finding container 92517de304b0faf30ddc0ba0b9b1de046caf061136510b8f3e22b947c0a838f2: Status 404 returned error can't find the container with id 92517de304b0faf30ddc0ba0b9b1de046caf061136510b8f3e22b947c0a838f2 Oct 30 13:40:57.877684 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:57.877639 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" event={"ID":"5287534f-89cd-4048-87a5-4d3039a9ac7d","Type":"ContainerStarted","Data":"92517de304b0faf30ddc0ba0b9b1de046caf061136510b8f3e22b947c0a838f2"} Oct 30 13:40:58.987450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:58.887947 5049 generic.go:356] "Generic (PLEG): container finished" podID="5287534f-89cd-4048-87a5-4d3039a9ac7d" containerID="6dc6944b56815f37423d8daa88acbd105866697adef974403ae65b2b4b82b5c4" exitCode=0 Oct 30 13:40:58.987450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:58.888030 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" event={"ID":"5287534f-89cd-4048-87a5-4d3039a9ac7d","Type":"ContainerDied","Data":"6dc6944b56815f37423d8daa88acbd105866697adef974403ae65b2b4b82b5c4"} Oct 30 13:40:59.145345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.139908 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:59.898271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.898182 5049 generic.go:356] "Generic (PLEG): container finished" podID="22a56f54-da40-419f-947b-34164430f3cc" containerID="51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e" exitCode=0 Oct 30 13:40:59.898271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.898253 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" event={"ID":"22a56f54-da40-419f-947b-34164430f3cc","Type":"ContainerDied","Data":"51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e"} Oct 30 13:40:59.898787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.898765 5049 scope.go:122] "RemoveContainer" containerID="51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e" Oct 30 13:40:59.900114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.900094 5049 generic.go:356] "Generic (PLEG): container finished" podID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerID="30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07" exitCode=0 Oct 30 13:40:59.900228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.900174 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" event={"ID":"46ab940e-48f1-4439-8842-e7bbba2b45b7","Type":"ContainerDied","Data":"30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07"} Oct 30 13:40:59.900654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.900636 5049 scope.go:122] "RemoveContainer" containerID="30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07" Oct 30 13:40:59.900901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.900797 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:59.900901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.900863 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:59.900901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.900897 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:59.903103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.903071 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:59.903201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.903152 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:59.903201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:40:59.903195 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:41:00.179776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.179749 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:41:00.335168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.335130 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5287534f-89cd-4048-87a5-4d3039a9ac7d-kube-api-access-clqrb\" (UniqueName: \"kubernetes.io/projected/5287534f-89cd-4048-87a5-4d3039a9ac7d-kube-api-access-clqrb\") pod \"5287534f-89cd-4048-87a5-4d3039a9ac7d\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " Oct 30 13:41:00.335427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.335279 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-openshift-ca\") pod \"5287534f-89cd-4048-87a5-4d3039a9ac7d\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " Oct 30 13:41:00.335526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.335514 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-xks-ca\") pod \"5287534f-89cd-4048-87a5-4d3039a9ac7d\" (UID: \"5287534f-89cd-4048-87a5-4d3039a9ac7d\") " Oct 30 13:41:00.335736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.335705 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-openshift-ca" pod "5287534f-89cd-4048-87a5-4d3039a9ac7d" (UID: "5287534f-89cd-4048-87a5-4d3039a9ac7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:00.336182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.336156 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:00.337635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.337613 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5287534f-89cd-4048-87a5-4d3039a9ac7d-kube-api-access-clqrb" pod "5287534f-89cd-4048-87a5-4d3039a9ac7d" (UID: "5287534f-89cd-4048-87a5-4d3039a9ac7d"). InnerVolumeSpecName "kube-api-access-clqrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:00.338267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.338246 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-xks-ca" pod "5287534f-89cd-4048-87a5-4d3039a9ac7d" (UID: "5287534f-89cd-4048-87a5-4d3039a9ac7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:00.377445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.377408 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:41:00.438070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.437968 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5287534f-89cd-4048-87a5-4d3039a9ac7d-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:00.438070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.438004 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clqrb\" (UniqueName: \"kubernetes.io/projected/5287534f-89cd-4048-87a5-4d3039a9ac7d-kube-api-access-clqrb\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:00.941408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.941364 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" event={"ID":"22a56f54-da40-419f-947b-34164430f3cc","Type":"ContainerStarted","Data":"a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1"} Oct 30 13:41:00.944521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.943716 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" event={"ID":"5287534f-89cd-4048-87a5-4d3039a9ac7d","Type":"ContainerDied","Data":"92517de304b0faf30ddc0ba0b9b1de046caf061136510b8f3e22b947c0a838f2"} Oct 30 13:41:00.944521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.943773 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92517de304b0faf30ddc0ba0b9b1de046caf061136510b8f3e22b947c0a838f2" Oct 30 13:41:00.944521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.943788 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt" Oct 30 13:41:00.949978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:00.949951 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" event={"ID":"46ab940e-48f1-4439-8842-e7bbba2b45b7","Type":"ContainerStarted","Data":"bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3"} Oct 30 13:41:01.218403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.209819 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:01.328344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.326493 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:41:01.328344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.326984 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 13:41:01.328344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.327035 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 13:41:01.496416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.496336 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj"] Oct 30 13:41:01.588555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.588514 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx"] Oct 30 13:41:01.698759 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.698732 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:01.773864 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.773767 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:41:01.888080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888040 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-envoy\") pod \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " Oct 30 13:41:01.888283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888153 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istiod-ca-cert\") pod \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " Oct 30 13:41:01.888283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888202 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-token\" (UniqueName: \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-token\") pod \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " Oct 30 13:41:01.888457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888349 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-certs\") pod \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " Oct 30 13:41:01.888457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888375 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-kube-api-access-8rqwf\" (UniqueName: \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-kube-api-access-8rqwf\") pod \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " Oct 30 13:41:01.888457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888427 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-data\") pod \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " Oct 30 13:41:01.888457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888444 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-socket\") pod \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " Oct 30 13:41:01.888689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888470 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-podinfo\") pod \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " Oct 30 13:41:01.888689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888527 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-credential-socket\") pod \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\" (UID: \"b4bcdc21-0f01-4986-a3d4-1a81611835f4\") " Oct 30 13:41:01.888689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888635 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-certs" pod "b4bcdc21-0f01-4986-a3d4-1a81611835f4" (UID: "b4bcdc21-0f01-4986-a3d4-1a81611835f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:01.888861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.888813 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-data" pod "b4bcdc21-0f01-4986-a3d4-1a81611835f4" (UID: "b4bcdc21-0f01-4986-a3d4-1a81611835f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:01.889194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.889160 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-socket" pod "b4bcdc21-0f01-4986-a3d4-1a81611835f4" (UID: "b4bcdc21-0f01-4986-a3d4-1a81611835f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:01.889194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.889181 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istiod-ca-cert" pod "b4bcdc21-0f01-4986-a3d4-1a81611835f4" (UID: "b4bcdc21-0f01-4986-a3d4-1a81611835f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:01.889496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.889472 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:01.889579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.889503 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:01.889579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.889517 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:01.889579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.889531 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:01.890171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.890144 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-credential-socket" pod "b4bcdc21-0f01-4986-a3d4-1a81611835f4" (UID: "b4bcdc21-0f01-4986-a3d4-1a81611835f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:01.891713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.891687 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-envoy" pod "b4bcdc21-0f01-4986-a3d4-1a81611835f4" (UID: "b4bcdc21-0f01-4986-a3d4-1a81611835f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:01.892068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.892018 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-podinfo" pod "b4bcdc21-0f01-4986-a3d4-1a81611835f4" (UID: "b4bcdc21-0f01-4986-a3d4-1a81611835f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:01.892160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.892087 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-kube-api-access-8rqwf" pod "b4bcdc21-0f01-4986-a3d4-1a81611835f4" (UID: "b4bcdc21-0f01-4986-a3d4-1a81611835f4"). InnerVolumeSpecName "kube-api-access-8rqwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:01.892160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.892128 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-token" pod "b4bcdc21-0f01-4986-a3d4-1a81611835f4" (UID: "b4bcdc21-0f01-4986-a3d4-1a81611835f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:01.934882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.918383 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:41:01.958346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.935179 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 13:41:01.974403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.973427 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 13:41:01.985694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.985655 5049 generic.go:356] "Generic (PLEG): container finished" podID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerID="10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c" exitCode=0 Oct 30 13:41:01.985910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.985885 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" event={"ID":"b4bcdc21-0f01-4986-a3d4-1a81611835f4","Type":"ContainerDied","Data":"10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c"} Oct 30 13:41:01.985993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.985952 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" event={"ID":"b4bcdc21-0f01-4986-a3d4-1a81611835f4","Type":"ContainerDied","Data":"799c51382a3667e7d7847ba15277a91b3509c0f998a0a3112660a1ca955d8d01"} Oct 30 13:41:01.985993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.985986 5049 scope.go:122] "RemoveContainer" containerID="10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c" Oct 30 13:41:01.987656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.987455 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b" Oct 30 13:41:01.992264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.992237 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rqwf\" (UniqueName: \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-kube-api-access-8rqwf\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:01.992461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.992266 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:01.992461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.992284 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:01.992461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.992298 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:01.992461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:01.992311 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bcdc21-0f01-4986-a3d4-1a81611835f4-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:02.005298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.005273 5049 scope.go:122] "RemoveContainer" containerID="ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8" Oct 30 13:41:02.077460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.077429 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b"] Oct 30 13:41:02.081717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.081690 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-4cb2b"] Oct 30 13:41:02.083472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.083404 5049 scope.go:122] "RemoveContainer" containerID="10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c" Oct 30 13:41:02.083846 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:02.083825 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c\": container with ID starting with 10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c not found: ID does not exist" containerID="10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c" Oct 30 13:41:02.083896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.083856 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c"} err="failed to get container status \"10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c\": rpc error: code = NotFound desc = could not find container \"10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c\": container with ID starting with 10459e1d927571a9dafd29f1dd5615d67545e84e21387617be03030b5989da9c not found: ID does not exist" Oct 30 13:41:02.083896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.083877 5049 scope.go:122] "RemoveContainer" containerID="ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8" Oct 30 13:41:02.084136 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:02.084118 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8\": container with ID starting with ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8 not found: ID does not exist" containerID="ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8" Oct 30 13:41:02.084174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.084150 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8"} err="failed to get container status \"ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8\": rpc error: code = NotFound desc = could not find container \"ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8\": container with ID starting with ee35be63d4171c637ebe61450266b8f90d85a6de278b1608865723b67e3166b8 not found: ID does not exist" Oct 30 13:41:02.319823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.316820 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 13:41:02.319823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.316909 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 13:41:02.924346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.915767 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 13:41:02.924346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:02.915831 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 13:41:03.108347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:03.103144 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:03.336346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:03.322387 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 13:41:03.336346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:03.322460 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 13:41:03.380004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:03.379969 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" path="/var/lib/kubelet/pods/b4bcdc21-0f01-4986-a3d4-1a81611835f4/volumes" Oct 30 13:41:03.921347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:03.915801 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 13:41:03.921347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:03.915879 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 13:41:04.329347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:04.328757 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 13:41:04.329347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:04.328832 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 13:41:04.774405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:04.774141 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:04.917252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:04.917208 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 13:41:04.917546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:04.917510 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 13:41:05.283791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:05.272493 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:05.328347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:05.317454 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 13:41:05.328347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:05.317533 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 13:41:05.920346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:05.914978 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 13:41:05.920346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:05.915064 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 13:41:06.316020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:06.315982 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Oct 30 13:41:06.316334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:06.316286 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Oct 30 13:41:06.921587 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:06.921542 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:41:06.923209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:06.923190 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:41:06.929967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:06.928488 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:41:07.284289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.284209 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:07.321947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.321915 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:41:07.322374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.322348 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:41:07.323119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.323098 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:41:07.438484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.438454 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:07.561762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.561731 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh"] Oct 30 13:41:07.561969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.561945 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" podUID="b8c15b62-e6fc-4b63-956b-41c266d073dc" containerName="maas-api" containerID="cri-o://dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91" gracePeriod=30 Oct 30 13:41:07.671963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.671920 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:07.896914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.896881 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:41:07.899832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.899805 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:07.931377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.931340 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-metrics-certs\") pod \"b8c15b62-e6fc-4b63-956b-41c266d073dc\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " Oct 30 13:41:07.931873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.931396 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c15b62-e6fc-4b63-956b-41c266d073dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c15b62-e6fc-4b63-956b-41c266d073dc-openshift-service-ca\") pod \"b8c15b62-e6fc-4b63-956b-41c266d073dc\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " Oct 30 13:41:07.931873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.931575 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-maas-api-tls\") pod \"b8c15b62-e6fc-4b63-956b-41c266d073dc\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " Oct 30 13:41:07.931873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.931760 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c15b62-e6fc-4b63-956b-41c266d073dc-kube-api-access-n872h\" (UniqueName: \"kubernetes.io/projected/b8c15b62-e6fc-4b63-956b-41c266d073dc-kube-api-access-n872h\") pod \"b8c15b62-e6fc-4b63-956b-41c266d073dc\" (UID: \"b8c15b62-e6fc-4b63-956b-41c266d073dc\") " Oct 30 13:41:07.931873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.931796 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c15b62-e6fc-4b63-956b-41c266d073dc-openshift-service-ca" pod "b8c15b62-e6fc-4b63-956b-41c266d073dc" (UID: "b8c15b62-e6fc-4b63-956b-41c266d073dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:07.932513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.932485 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c15b62-e6fc-4b63-956b-41c266d073dc-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:07.933657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.933632 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-metrics-certs" pod "b8c15b62-e6fc-4b63-956b-41c266d073dc" (UID: "b8c15b62-e6fc-4b63-956b-41c266d073dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:07.933778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.933698 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-maas-api-tls" pod "b8c15b62-e6fc-4b63-956b-41c266d073dc" (UID: "b8c15b62-e6fc-4b63-956b-41c266d073dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:07.933955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:07.933927 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c15b62-e6fc-4b63-956b-41c266d073dc-kube-api-access-n872h" pod "b8c15b62-e6fc-4b63-956b-41c266d073dc" (UID: "b8c15b62-e6fc-4b63-956b-41c266d073dc"). InnerVolumeSpecName "kube-api-access-n872h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:08.034199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.034162 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:08.034199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.034201 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c15b62-e6fc-4b63-956b-41c266d073dc-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:08.034426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.034216 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n872h\" (UniqueName: \"kubernetes.io/projected/b8c15b62-e6fc-4b63-956b-41c266d073dc-kube-api-access-n872h\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:08.054781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.054749 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:08.301596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.301556 5049 generic.go:356] "Generic (PLEG): container finished" podID="b8c15b62-e6fc-4b63-956b-41c266d073dc" containerID="dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91" exitCode=0 Oct 30 13:41:08.302014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.301981 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" event={"ID":"b8c15b62-e6fc-4b63-956b-41c266d073dc","Type":"ContainerDied","Data":"dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91"} Oct 30 13:41:08.302103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.302084 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" event={"ID":"b8c15b62-e6fc-4b63-956b-41c266d073dc","Type":"ContainerDied","Data":"37d3263688d06605eef415e24f938a5bfe4d2c7b4de496cd51aebe21d87ce1c0"} Oct 30 13:41:08.302170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.302126 5049 scope.go:122] "RemoveContainer" containerID="dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91" Oct 30 13:41:08.302413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.302395 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh" Oct 30 13:41:08.329904 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.329874 5049 scope.go:122] "RemoveContainer" containerID="dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91" Oct 30 13:41:08.330746 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:08.330508 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91\": container with ID starting with dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91 not found: ID does not exist" containerID="dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91" Oct 30 13:41:08.330746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.330549 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91"} err="failed to get container status \"dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91\": rpc error: code = NotFound desc = could not find container \"dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91\": container with ID starting with dd8d1d67af014cc6cbf215f7fc4ce63074b6ea26089b80b264c33c404c65fa91 not found: ID does not exist" Oct 30 13:41:08.373173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.373139 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh"] Oct 30 13:41:08.381225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.381192 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-428a0e-7555dfcff9-n9lsh"] Oct 30 13:41:08.433390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.429091 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:08.702470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.702366 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:08.705797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.705762 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r"] Oct 30 13:41:08.716574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.716544 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-428a0e-m9f5r"] Oct 30 13:41:08.911372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:08.903141 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:09.230076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:09.230042 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:09.232227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:09.232205 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8"] Oct 30 13:41:09.276261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:09.276220 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd"] Oct 30 13:41:09.344031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:09.343989 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d542fa-4886-4c68-a55f-19c1fbee1837" path="/var/lib/kubelet/pods/72d542fa-4886-4c68-a55f-19c1fbee1837/volumes" Oct 30 13:41:09.344659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:09.344637 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c15b62-e6fc-4b63-956b-41c266d073dc" path="/var/lib/kubelet/pods/b8c15b62-e6fc-4b63-956b-41c266d073dc/volumes" Oct 30 13:41:09.431070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:09.431033 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:09.712848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:09.712812 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:09.976100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:09.967940 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:11.796988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:11.796954 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:13.711389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:13.706852 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:14.232680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.232638 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" podUID="135ac821-40f8-46bf-a3c4-f133dd61053e" containerName="payload-processing" containerID="cri-o://92a1a787a7fa06a6f5955a9d7bbced3f17e856cf19e65ccc44da6ae5162a08bf" gracePeriod=25 Oct 30 13:41:14.277553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.277518 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" podUID="30ee1fb6-0d37-44ee-b530-886fd8ad2fc9" containerName="payload-pre-processing" containerID="cri-o://8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949" gracePeriod=25 Oct 30 13:41:14.370335 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.370293 5049 generic.go:356] "Generic (PLEG): container finished" podID="135ac821-40f8-46bf-a3c4-f133dd61053e" containerID="92a1a787a7fa06a6f5955a9d7bbced3f17e856cf19e65ccc44da6ae5162a08bf" exitCode=0 Oct 30 13:41:14.370507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.370352 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" event={"ID":"135ac821-40f8-46bf-a3c4-f133dd61053e","Type":"ContainerDied","Data":"92a1a787a7fa06a6f5955a9d7bbced3f17e856cf19e65ccc44da6ae5162a08bf"} Oct 30 13:41:14.587145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.587111 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:41:14.628394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.628282 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:41:14.674913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.673986 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135ac821-40f8-46bf-a3c4-f133dd61053e-kube-api-access-ww2qn\" (UniqueName: \"kubernetes.io/projected/135ac821-40f8-46bf-a3c4-f133dd61053e-kube-api-access-ww2qn\") pod \"135ac821-40f8-46bf-a3c4-f133dd61053e\" (UID: \"135ac821-40f8-46bf-a3c4-f133dd61053e\") " Oct 30 13:41:14.674913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.674378 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135ac821-40f8-46bf-a3c4-f133dd61053e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135ac821-40f8-46bf-a3c4-f133dd61053e-plugins-config-volume\") pod \"135ac821-40f8-46bf-a3c4-f133dd61053e\" (UID: \"135ac821-40f8-46bf-a3c4-f133dd61053e\") " Oct 30 13:41:14.674913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.674809 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135ac821-40f8-46bf-a3c4-f133dd61053e-plugins-config-volume" pod "135ac821-40f8-46bf-a3c4-f133dd61053e" (UID: "135ac821-40f8-46bf-a3c4-f133dd61053e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:14.675279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.675257 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135ac821-40f8-46bf-a3c4-f133dd61053e-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:14.679096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.676879 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135ac821-40f8-46bf-a3c4-f133dd61053e-kube-api-access-ww2qn" pod "135ac821-40f8-46bf-a3c4-f133dd61053e" (UID: "135ac821-40f8-46bf-a3c4-f133dd61053e"). InnerVolumeSpecName "kube-api-access-ww2qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:14.776428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.776343 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-plugins-config-volume\") pod \"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9\" (UID: \"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9\") " Oct 30 13:41:14.776847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.776573 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-kube-api-access-l5j2j\" (UniqueName: \"kubernetes.io/projected/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-kube-api-access-l5j2j\") pod \"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9\" (UID: \"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9\") " Oct 30 13:41:14.776847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.776626 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-plugins-config-volume" pod "30ee1fb6-0d37-44ee-b530-886fd8ad2fc9" (UID: "30ee1fb6-0d37-44ee-b530-886fd8ad2fc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:14.777233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.777215 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww2qn\" (UniqueName: \"kubernetes.io/projected/135ac821-40f8-46bf-a3c4-f133dd61053e-kube-api-access-ww2qn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:14.777276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.777238 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:14.778687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.778659 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-kube-api-access-l5j2j" pod "30ee1fb6-0d37-44ee-b530-886fd8ad2fc9" (UID: "30ee1fb6-0d37-44ee-b530-886fd8ad2fc9"). InnerVolumeSpecName "kube-api-access-l5j2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:14.878428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:14.878394 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5j2j\" (UniqueName: \"kubernetes.io/projected/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9-kube-api-access-l5j2j\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:15.379561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.379533 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" Oct 30 13:41:15.379728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.379573 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8" event={"ID":"135ac821-40f8-46bf-a3c4-f133dd61053e","Type":"ContainerDied","Data":"2a8c883e6c080423ff70cc69ace52ad684cd90dc98441500249e515b9c5e83f2"} Oct 30 13:41:15.379728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.379643 5049 scope.go:122] "RemoveContainer" containerID="92a1a787a7fa06a6f5955a9d7bbced3f17e856cf19e65ccc44da6ae5162a08bf" Oct 30 13:41:15.381334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.381295 5049 generic.go:356] "Generic (PLEG): container finished" podID="30ee1fb6-0d37-44ee-b530-886fd8ad2fc9" containerID="8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949" exitCode=0 Oct 30 13:41:15.382197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.381354 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" Oct 30 13:41:15.382197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.382100 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" event={"ID":"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9","Type":"ContainerDied","Data":"8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949"} Oct 30 13:41:15.382197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.382141 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd" event={"ID":"30ee1fb6-0d37-44ee-b530-886fd8ad2fc9","Type":"ContainerDied","Data":"48d0539f43229e93d2ea8ae046911c330046c9591627eee6070556ee738a7f8d"} Oct 30 13:41:15.392700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.392679 5049 scope.go:122] "RemoveContainer" containerID="8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949" Oct 30 13:41:15.403557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.403532 5049 scope.go:122] "RemoveContainer" containerID="8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949" Oct 30 13:41:15.403879 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:15.403857 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949\": container with ID starting with 8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949 not found: ID does not exist" containerID="8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949" Oct 30 13:41:15.403977 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.403885 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949"} err="failed to get container status \"8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949\": rpc error: code = NotFound desc = could not find container \"8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949\": container with ID starting with 8a815dc3b7bdae3202d14e594efd73738306bd37e83433c87daa081403fcc949 not found: ID does not exist" Oct 30 13:41:15.431397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.431366 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8"] Oct 30 13:41:15.441137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.441109 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6b9f4c9594-rkhp8"] Oct 30 13:41:15.471458 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.471423 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd"] Oct 30 13:41:15.477194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.477162 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-698695df9-sl9hd"] Oct 30 13:41:15.573236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:15.573200 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:16.256469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:16.256011 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:16.422612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:16.422579 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:16.511861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:16.511771 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj"] Oct 30 13:41:16.512055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:16.512025 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" containerID="cri-o://127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf" gracePeriod=30 Oct 30 13:41:16.591419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:16.591388 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:16.618553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:16.618506 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" containerID="cri-o://33b4c95539c43d800e7e55634883ca697cfab7ffb5b823bd2882d7d8ec60a7f7" gracePeriod=45 Oct 30 13:41:16.712975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:16.710499 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" containerID="cri-o://1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625" gracePeriod=45 Oct 30 13:41:16.917689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:16.917656 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": dial tcp 10.132.0.107:8000: connect: connection refused" Oct 30 13:41:17.038557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.038525 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:41:17.115426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.115390 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-dshm\") pod \"f7747aab-1a05-402a-b859-9d4afa588683\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " Oct 30 13:41:17.115633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.115485 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-model-cache\") pod \"f7747aab-1a05-402a-b859-9d4afa588683\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " Oct 30 13:41:17.115701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.115652 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7747aab-1a05-402a-b859-9d4afa588683-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7747aab-1a05-402a-b859-9d4afa588683-tls-certs\") pod \"f7747aab-1a05-402a-b859-9d4afa588683\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " Oct 30 13:41:17.116163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.115821 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-tmp-dir\") pod \"f7747aab-1a05-402a-b859-9d4afa588683\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " Oct 30 13:41:17.116163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.115882 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-home\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-home\") pod \"f7747aab-1a05-402a-b859-9d4afa588683\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " Oct 30 13:41:17.116163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.115954 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7747aab-1a05-402a-b859-9d4afa588683-kube-api-access-bfqjh\" (UniqueName: \"kubernetes.io/projected/f7747aab-1a05-402a-b859-9d4afa588683-kube-api-access-bfqjh\") pod \"f7747aab-1a05-402a-b859-9d4afa588683\" (UID: \"f7747aab-1a05-402a-b859-9d4afa588683\") " Oct 30 13:41:17.117964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.117933 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-tmp-dir" pod "f7747aab-1a05-402a-b859-9d4afa588683" (UID: "f7747aab-1a05-402a-b859-9d4afa588683"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.117964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.117948 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-model-cache" pod "f7747aab-1a05-402a-b859-9d4afa588683" (UID: "f7747aab-1a05-402a-b859-9d4afa588683"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.118197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.118061 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-home" pod "f7747aab-1a05-402a-b859-9d4afa588683" (UID: "f7747aab-1a05-402a-b859-9d4afa588683"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.121705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.121659 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7747aab-1a05-402a-b859-9d4afa588683-kube-api-access-bfqjh" pod "f7747aab-1a05-402a-b859-9d4afa588683" (UID: "f7747aab-1a05-402a-b859-9d4afa588683"). InnerVolumeSpecName "kube-api-access-bfqjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:17.121705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.121668 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7747aab-1a05-402a-b859-9d4afa588683-tls-certs" pod "f7747aab-1a05-402a-b859-9d4afa588683" (UID: "f7747aab-1a05-402a-b859-9d4afa588683"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:17.122271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.122245 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-dshm" pod "f7747aab-1a05-402a-b859-9d4afa588683" (UID: "f7747aab-1a05-402a-b859-9d4afa588683"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.218733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.218697 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.218733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.218730 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.218949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.218742 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfqjh\" (UniqueName: \"kubernetes.io/projected/f7747aab-1a05-402a-b859-9d4afa588683-kube-api-access-bfqjh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.218949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.218753 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.218949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.218762 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7747aab-1a05-402a-b859-9d4afa588683-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.218949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.218771 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7747aab-1a05-402a-b859-9d4afa588683-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.339266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.339223 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135ac821-40f8-46bf-a3c4-f133dd61053e" path="/var/lib/kubelet/pods/135ac821-40f8-46bf-a3c4-f133dd61053e/volumes" Oct 30 13:41:17.339952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.339930 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ee1fb6-0d37-44ee-b530-886fd8ad2fc9" path="/var/lib/kubelet/pods/30ee1fb6-0d37-44ee-b530-886fd8ad2fc9/volumes" Oct 30 13:41:17.415497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.415464 5049 generic.go:356] "Generic (PLEG): container finished" podID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerID="33b4c95539c43d800e7e55634883ca697cfab7ffb5b823bd2882d7d8ec60a7f7" exitCode=0 Oct 30 13:41:17.415753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.415539 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" event={"ID":"a22fa542-32e1-4074-9d28-cadb02442c7b","Type":"ContainerDied","Data":"33b4c95539c43d800e7e55634883ca697cfab7ffb5b823bd2882d7d8ec60a7f7"} Oct 30 13:41:17.417513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.417485 5049 generic.go:356] "Generic (PLEG): container finished" podID="f7747aab-1a05-402a-b859-9d4afa588683" containerID="1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625" exitCode=0 Oct 30 13:41:17.417638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.417561 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" Oct 30 13:41:17.417638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.417572 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" event={"ID":"f7747aab-1a05-402a-b859-9d4afa588683","Type":"ContainerDied","Data":"1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625"} Oct 30 13:41:17.417796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.417650 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx" event={"ID":"f7747aab-1a05-402a-b859-9d4afa588683","Type":"ContainerDied","Data":"b28ce877ebebf587a3872ea7636845e01efddc6dfce46c2d0cbce26201f33905"} Oct 30 13:41:17.417796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.417695 5049 scope.go:122] "RemoveContainer" containerID="1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625" Oct 30 13:41:17.432131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.432105 5049 scope.go:122] "RemoveContainer" containerID="1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625" Oct 30 13:41:17.432601 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:17.432573 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625\": container with ID starting with 1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625 not found: ID does not exist" containerID="1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625" Oct 30 13:41:17.432722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.432608 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625"} err="failed to get container status \"1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625\": rpc error: code = NotFound desc = could not find container \"1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625\": container with ID starting with 1cd6e160f9131ea1234fba8e253482dbbaaf2ee732b790866a0094e3a1e93625 not found: ID does not exist" Oct 30 13:41:17.465047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.465017 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:41:17.473208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.473177 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx"] Oct 30 13:41:17.477184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.477159 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/simulator-w2-cb58b8-kserve-7dcfb478f5-95pqx"] Oct 30 13:41:17.522357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.522288 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-tmp-dir\") pod \"a22fa542-32e1-4074-9d28-cadb02442c7b\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " Oct 30 13:41:17.522625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.522596 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-tmp-dir" pod "a22fa542-32e1-4074-9d28-cadb02442c7b" (UID: "a22fa542-32e1-4074-9d28-cadb02442c7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.522753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.522601 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-home\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-home\") pod \"a22fa542-32e1-4074-9d28-cadb02442c7b\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " Oct 30 13:41:17.522753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.522691 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22fa542-32e1-4074-9d28-cadb02442c7b-kube-api-access-hmtml\" (UniqueName: \"kubernetes.io/projected/a22fa542-32e1-4074-9d28-cadb02442c7b-kube-api-access-hmtml\") pod \"a22fa542-32e1-4074-9d28-cadb02442c7b\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " Oct 30 13:41:17.522832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.522766 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-model-cache\") pod \"a22fa542-32e1-4074-9d28-cadb02442c7b\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " Oct 30 13:41:17.522878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.522853 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-dshm\") pod \"a22fa542-32e1-4074-9d28-cadb02442c7b\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " Oct 30 13:41:17.523141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.522919 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22fa542-32e1-4074-9d28-cadb02442c7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a22fa542-32e1-4074-9d28-cadb02442c7b-tls-certs\") pod \"a22fa542-32e1-4074-9d28-cadb02442c7b\" (UID: \"a22fa542-32e1-4074-9d28-cadb02442c7b\") " Oct 30 13:41:17.523249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.523167 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-home" pod "a22fa542-32e1-4074-9d28-cadb02442c7b" (UID: "a22fa542-32e1-4074-9d28-cadb02442c7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.523831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.523395 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-model-cache" pod "a22fa542-32e1-4074-9d28-cadb02442c7b" (UID: "a22fa542-32e1-4074-9d28-cadb02442c7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.523831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.523769 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.523831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.523793 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.523831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.523808 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.525892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.525860 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-dshm" pod "a22fa542-32e1-4074-9d28-cadb02442c7b" (UID: "a22fa542-32e1-4074-9d28-cadb02442c7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.526306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.526274 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22fa542-32e1-4074-9d28-cadb02442c7b-tls-certs" pod "a22fa542-32e1-4074-9d28-cadb02442c7b" (UID: "a22fa542-32e1-4074-9d28-cadb02442c7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:17.526824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.526801 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22fa542-32e1-4074-9d28-cadb02442c7b-kube-api-access-hmtml" pod "a22fa542-32e1-4074-9d28-cadb02442c7b" (UID: "a22fa542-32e1-4074-9d28-cadb02442c7b"). InnerVolumeSpecName "kube-api-access-hmtml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:17.554175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.554085 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:17.649929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.637613 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmtml\" (UniqueName: \"kubernetes.io/projected/a22fa542-32e1-4074-9d28-cadb02442c7b-kube-api-access-hmtml\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.652484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.652456 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22fa542-32e1-4074-9d28-cadb02442c7b-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.652693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:17.652673 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22fa542-32e1-4074-9d28-cadb02442c7b-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:18.430399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:18.430367 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" Oct 30 13:41:18.430399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:18.430385 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj" event={"ID":"a22fa542-32e1-4074-9d28-cadb02442c7b","Type":"ContainerDied","Data":"0764258fa4922cf4eb330e749ce41bc8d917025ac3b0f1a6fe4a713b601cda59"} Oct 30 13:41:18.430913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:18.430447 5049 scope.go:122] "RemoveContainer" containerID="33b4c95539c43d800e7e55634883ca697cfab7ffb5b823bd2882d7d8ec60a7f7" Oct 30 13:41:18.478819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:18.478791 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj"] Oct 30 13:41:18.488560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:18.488533 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-w2-cb58b8-kserve-7696cf655c-7xzvj"] Oct 30 13:41:18.872582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:18.872546 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:19.053192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.053154 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:19.285279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.285190 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5"] Oct 30 13:41:19.290744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.290709 5049 cpu_manager.go:414] "containerMap: removing container" podUID="135ac821-40f8-46bf-a3c4-f133dd61053e" containerName="payload-processing" Oct 30 13:41:19.290952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.290936 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135ac821-40f8-46bf-a3c4-f133dd61053e" containerName="payload-processing" Oct 30 13:41:19.291070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291059 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b8c15b62-e6fc-4b63-956b-41c266d073dc" containerName="maas-api" Oct 30 13:41:19.291153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291142 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c15b62-e6fc-4b63-956b-41c266d073dc" containerName="maas-api" Oct 30 13:41:19.291261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291250 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" Oct 30 13:41:19.291361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291349 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" Oct 30 13:41:19.291462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291450 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5287534f-89cd-4048-87a5-4d3039a9ac7d" containerName="revoke-keys" Oct 30 13:41:19.291543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291532 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5287534f-89cd-4048-87a5-4d3039a9ac7d" containerName="revoke-keys" Oct 30 13:41:19.291652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291639 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" Oct 30 13:41:19.291740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291729 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" Oct 30 13:41:19.291836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291827 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" Oct 30 13:41:19.291906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291897 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" Oct 30 13:41:19.291991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.291981 5049 cpu_manager.go:414] "containerMap: removing container" podUID="30ee1fb6-0d37-44ee-b530-886fd8ad2fc9" containerName="payload-pre-processing" Oct 30 13:41:19.292065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.292054 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ee1fb6-0d37-44ee-b530-886fd8ad2fc9" containerName="payload-pre-processing" Oct 30 13:41:19.292138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.292130 5049 cpu_manager.go:414] "containerMap: removing container" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" Oct 30 13:41:19.292202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.292193 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" Oct 30 13:41:19.292517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.292503 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" Oct 30 13:41:19.292611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.292603 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4bcdc21-0f01-4986-a3d4-1a81611835f4" containerName="istio-proxy" Oct 30 13:41:19.292715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.292704 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ee1fb6-0d37-44ee-b530-886fd8ad2fc9" containerName="payload-pre-processing" Oct 30 13:41:19.292781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.292773 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" containerName="main" Oct 30 13:41:19.292853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.292846 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="135ac821-40f8-46bf-a3c4-f133dd61053e" containerName="payload-processing" Oct 30 13:41:19.292922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.292915 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c15b62-e6fc-4b63-956b-41c266d073dc" containerName="maas-api" Oct 30 13:41:19.292990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.292982 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7747aab-1a05-402a-b859-9d4afa588683" containerName="main" Oct 30 13:41:19.293054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.293045 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5287534f-89cd-4048-87a5-4d3039a9ac7d" containerName="revoke-keys" Oct 30 13:41:19.314343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.309246 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:41:19.330381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.326748 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-30c004\"" Oct 30 13:41:19.423532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.423480 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22fa542-32e1-4074-9d28-cadb02442c7b" path="/var/lib/kubelet/pods/a22fa542-32e1-4074-9d28-cadb02442c7b/volumes" Oct 30 13:41:19.425768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.425628 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7747aab-1a05-402a-b859-9d4afa588683" path="/var/lib/kubelet/pods/f7747aab-1a05-402a-b859-9d4afa588683/volumes" Oct 30 13:41:19.467341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.457677 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5"] Oct 30 13:41:19.471903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.471586 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6"] Oct 30 13:41:19.478387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.477777 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6"] Oct 30 13:41:19.478387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.477892 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:41:19.491897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.491870 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-30c004-dockercfg-9jcrt\"" Oct 30 13:41:19.530416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.530371 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692z5\" (UniqueName: \"kubernetes.io/projected/c5a91a94-c1e6-4afc-8576-631b1361582f-kube-api-access-692z5\") pod \"payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5\" (UID: \"c5a91a94-c1e6-4afc-8576-631b1361582f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:41:19.530575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.530550 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4vcl\" (UniqueName: \"kubernetes.io/projected/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-kube-api-access-s4vcl\") pod \"payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6\" (UID: \"647f2071-e66f-4f37-bff7-6bbfb60dd5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:41:19.530663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.530650 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6\" (UID: \"647f2071-e66f-4f37-bff7-6bbfb60dd5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:41:19.530732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.530719 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a91a94-c1e6-4afc-8576-631b1361582f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5\" (UID: \"c5a91a94-c1e6-4afc-8576-631b1361582f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:41:19.632131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.632089 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6\" (UID: \"647f2071-e66f-4f37-bff7-6bbfb60dd5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:41:19.632363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.632222 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a91a94-c1e6-4afc-8576-631b1361582f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5\" (UID: \"c5a91a94-c1e6-4afc-8576-631b1361582f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:41:19.632438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.632400 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692z5\" (UniqueName: \"kubernetes.io/projected/c5a91a94-c1e6-4afc-8576-631b1361582f-kube-api-access-692z5\") pod \"payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5\" (UID: \"c5a91a94-c1e6-4afc-8576-631b1361582f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:41:19.632550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.632534 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vcl\" (UniqueName: \"kubernetes.io/projected/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-kube-api-access-s4vcl\") pod \"payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6\" (UID: \"647f2071-e66f-4f37-bff7-6bbfb60dd5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:41:19.633075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.633050 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6\" (UID: \"647f2071-e66f-4f37-bff7-6bbfb60dd5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:41:19.633164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.633153 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a91a94-c1e6-4afc-8576-631b1361582f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5\" (UID: \"c5a91a94-c1e6-4afc-8576-631b1361582f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:41:19.654445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.654410 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-692z5\" (UniqueName: \"kubernetes.io/projected/c5a91a94-c1e6-4afc-8576-631b1361582f-kube-api-access-692z5\") pod \"payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5\" (UID: \"c5a91a94-c1e6-4afc-8576-631b1361582f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:41:19.654621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.654427 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vcl\" (UniqueName: \"kubernetes.io/projected/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-kube-api-access-s4vcl\") pod \"payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6\" (UID: \"647f2071-e66f-4f37-bff7-6bbfb60dd5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:41:19.687553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.687523 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:41:19.794383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:19.793998 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:41:20.011642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:20.011599 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5"] Oct 30 13:41:20.015802 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:41:20.015776 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a91a94_c1e6_4afc_8576_631b1361582f.slice/crio-567a4064a065d80f15dd0c055581954b2b006750925335bc75271eddfb5a589b WatchSource:0}: Error finding container 567a4064a065d80f15dd0c055581954b2b006750925335bc75271eddfb5a589b: Status 404 returned error can't find the container with id 567a4064a065d80f15dd0c055581954b2b006750925335bc75271eddfb5a589b Oct 30 13:41:20.018126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:20.018106 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:41:20.131550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:20.131516 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6"] Oct 30 13:41:20.134274 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:41:20.134217 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647f2071_e66f_4f37_bff7_6bbfb60dd5c1.slice/crio-8e627e9ff2e0cc582ec57a3e973b2128ed7148c8a47a6e0d3d845a9ae70d516e WatchSource:0}: Error finding container 8e627e9ff2e0cc582ec57a3e973b2128ed7148c8a47a6e0d3d845a9ae70d516e: Status 404 returned error can't find the container with id 8e627e9ff2e0cc582ec57a3e973b2128ed7148c8a47a6e0d3d845a9ae70d516e Oct 30 13:41:20.159591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:20.159549 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:20.494783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:20.494694 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" event={"ID":"647f2071-e66f-4f37-bff7-6bbfb60dd5c1","Type":"ContainerStarted","Data":"8e627e9ff2e0cc582ec57a3e973b2128ed7148c8a47a6e0d3d845a9ae70d516e"} Oct 30 13:41:20.496596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:20.496563 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" event={"ID":"c5a91a94-c1e6-4afc-8576-631b1361582f","Type":"ContainerStarted","Data":"567a4064a065d80f15dd0c055581954b2b006750925335bc75271eddfb5a589b"} Oct 30 13:41:20.513944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:20.513908 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:41:20.514114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:20.513969 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:41:21.772926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.772869 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6"] Oct 30 13:41:21.782441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.782343 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:21.806470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.806442 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:41:21.815723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.815694 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:41:21.831350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.831268 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6"] Oct 30 13:41:21.869656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.869623 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tnpj\" (UniqueName: \"kubernetes.io/projected/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-kube-api-access-2tnpj\") pod \"maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:21.869833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.869773 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:21.869833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.869797 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:21.879962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.879936 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:41:21.971468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.971435 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-socket\") pod \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " Oct 30 13:41:21.971669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.971480 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istiod-ca-cert\") pod \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " Oct 30 13:41:21.971669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.971513 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-credential-socket\") pod \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " Oct 30 13:41:21.971669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.971589 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-podinfo\") pod \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " Oct 30 13:41:21.971669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.971647 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-certs\") pod \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " Oct 30 13:41:21.971913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.971718 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-envoy\") pod \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " Oct 30 13:41:21.971913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.971791 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-token\" (UniqueName: \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-token\") pod \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " Oct 30 13:41:21.971913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.971812 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-kube-api-access-dk86p\" (UniqueName: \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-kube-api-access-dk86p\") pod \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " Oct 30 13:41:21.971913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.971857 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-data\") pod \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\" (UID: \"c491c8a2-4e07-4851-aa5d-6a55dc247c59\") " Oct 30 13:41:21.972139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.971989 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:21.972139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.972013 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:21.972210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.972172 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tnpj\" (UniqueName: \"kubernetes.io/projected/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-kube-api-access-2tnpj\") pod \"maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:21.973108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.973078 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-socket" pod "c491c8a2-4e07-4851-aa5d-6a55dc247c59" (UID: "c491c8a2-4e07-4851-aa5d-6a55dc247c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:21.974768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.974490 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-credential-socket" pod "c491c8a2-4e07-4851-aa5d-6a55dc247c59" (UID: "c491c8a2-4e07-4851-aa5d-6a55dc247c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:21.975258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.975042 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istiod-ca-cert" pod "c491c8a2-4e07-4851-aa5d-6a55dc247c59" (UID: "c491c8a2-4e07-4851-aa5d-6a55dc247c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:21.975447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.975400 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-certs" pod "c491c8a2-4e07-4851-aa5d-6a55dc247c59" (UID: "c491c8a2-4e07-4851-aa5d-6a55dc247c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:21.976047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.976026 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:21.976135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.976103 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-data" pod "c491c8a2-4e07-4851-aa5d-6a55dc247c59" (UID: "c491c8a2-4e07-4851-aa5d-6a55dc247c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:21.976655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.976607 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-envoy" pod "c491c8a2-4e07-4851-aa5d-6a55dc247c59" (UID: "c491c8a2-4e07-4851-aa5d-6a55dc247c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:21.979976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.979929 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-token" pod "c491c8a2-4e07-4851-aa5d-6a55dc247c59" (UID: "c491c8a2-4e07-4851-aa5d-6a55dc247c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:21.981902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.981580 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:21.982208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.982185 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-podinfo" pod "c491c8a2-4e07-4851-aa5d-6a55dc247c59" (UID: "c491c8a2-4e07-4851-aa5d-6a55dc247c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:21.982419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.982395 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-kube-api-access-dk86p" pod "c491c8a2-4e07-4851-aa5d-6a55dc247c59" (UID: "c491c8a2-4e07-4851-aa5d-6a55dc247c59"). InnerVolumeSpecName "kube-api-access-dk86p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:21.997456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:21.997406 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tnpj\" (UniqueName: \"kubernetes.io/projected/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-kube-api-access-2tnpj\") pod \"maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:22.076631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.076488 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:22.079043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.078217 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:22.079043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.078247 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:22.079043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.078261 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk86p\" (UniqueName: \"kubernetes.io/projected/c491c8a2-4e07-4851-aa5d-6a55dc247c59-kube-api-access-dk86p\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:22.079043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.078275 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:22.079043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.078289 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:22.079043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.078303 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:22.079043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.078316 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:22.079043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.078345 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c491c8a2-4e07-4851-aa5d-6a55dc247c59-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:22.079043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.078362 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c491c8a2-4e07-4851-aa5d-6a55dc247c59-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:22.130447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.130167 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:22.470233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.470201 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6"] Oct 30 13:41:22.477397 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:41:22.477292 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea9d9ea_4bd0_48da_b94a_a9a0db4e9852.slice/crio-d6ae28b6dc513b187a1b2e32c14abd382c7bf08110c606acaca64ca9410eb49f WatchSource:0}: Error finding container d6ae28b6dc513b187a1b2e32c14abd382c7bf08110c606acaca64ca9410eb49f: Status 404 returned error can't find the container with id d6ae28b6dc513b187a1b2e32c14abd382c7bf08110c606acaca64ca9410eb49f Oct 30 13:41:22.525037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.525001 5049 generic.go:356] "Generic (PLEG): container finished" podID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerID="127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf" exitCode=0 Oct 30 13:41:22.525191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.525101 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" event={"ID":"c491c8a2-4e07-4851-aa5d-6a55dc247c59","Type":"ContainerDied","Data":"127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf"} Oct 30 13:41:22.525191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.525151 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" event={"ID":"c491c8a2-4e07-4851-aa5d-6a55dc247c59","Type":"ContainerDied","Data":"25d40b0e1888c6cafb7d3271663c460206698c8b38a9759c0c52e2bc9ddb109c"} Oct 30 13:41:22.525191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.525183 5049 scope.go:122] "RemoveContainer" containerID="127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf" Oct 30 13:41:22.525380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.525357 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj" Oct 30 13:41:22.531739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.531708 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" event={"ID":"c5a91a94-c1e6-4afc-8576-631b1361582f","Type":"ContainerStarted","Data":"30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf"} Oct 30 13:41:22.532571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.532549 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:41:22.534245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.534220 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" event={"ID":"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852","Type":"ContainerStarted","Data":"d6ae28b6dc513b187a1b2e32c14abd382c7bf08110c606acaca64ca9410eb49f"} Oct 30 13:41:22.536125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.536098 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" event={"ID":"647f2071-e66f-4f37-bff7-6bbfb60dd5c1","Type":"ContainerStarted","Data":"98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5"} Oct 30 13:41:22.536886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.536867 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:41:22.551777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.551621 5049 scope.go:122] "RemoveContainer" containerID="2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5" Oct 30 13:41:22.564877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.564857 5049 scope.go:122] "RemoveContainer" containerID="127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf" Oct 30 13:41:22.565161 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:22.565137 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf\": container with ID starting with 127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf not found: ID does not exist" containerID="127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf" Oct 30 13:41:22.565251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.565172 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf"} err="failed to get container status \"127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf\": rpc error: code = NotFound desc = could not find container \"127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf\": container with ID starting with 127521eb75481a9299c06060a94c2fd42ff91a54af78f8fa5f2f37c302e637cf not found: ID does not exist" Oct 30 13:41:22.565251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.565194 5049 scope.go:122] "RemoveContainer" containerID="2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5" Oct 30 13:41:22.565499 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:22.565481 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5\": container with ID starting with 2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5 not found: ID does not exist" containerID="2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5" Oct 30 13:41:22.565546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.565508 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5"} err="failed to get container status \"2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5\": rpc error: code = NotFound desc = could not find container \"2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5\": container with ID starting with 2630180eebd88542829455172a0b41aa0931a4a6c0e338fffdf6942bad913ef5 not found: ID does not exist" Oct 30 13:41:22.582787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.582745 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" podStartSLOduration=1.318193506 podStartE2EDuration="3.582732671s" totalImagesPullingTime="2.264539165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:22.57909686 +0000 UTC m=+2989.394291284" watchObservedRunningTime="2026-10-30 13:41:22.582732671 +0000 UTC m=+2989.397927093" Oct 30 13:41:22.616059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.616012 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" podStartSLOduration=1.224531 podStartE2EDuration="3.615995002s" totalImagesPullingTime="2.391464002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:22.615128661 +0000 UTC m=+2989.430323084" watchObservedRunningTime="2026-10-30 13:41:22.615995002 +0000 UTC m=+2989.431189425" Oct 30 13:41:22.655150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.655061 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj"] Oct 30 13:41:22.659209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:22.659180 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-428a0e-openshift-default-6f49f4b67-f8qdj"] Oct 30 13:41:23.343267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:23.343231 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" path="/var/lib/kubelet/pods/c491c8a2-4e07-4851-aa5d-6a55dc247c59/volumes" Oct 30 13:41:23.549247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:23.549211 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" event={"ID":"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852","Type":"ContainerStarted","Data":"86fbe0dd8554a16f1181c3da13107a78c63ab31740701606ab0c5255a76aeeba"} Oct 30 13:41:23.589642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:23.589589 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" podStartSLOduration=2.589576438 podStartE2EDuration="2.589576438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:23.585579978 +0000 UTC m=+2990.400774402" watchObservedRunningTime="2026-10-30 13:41:23.589576438 +0000 UTC m=+2990.404770860" Oct 30 13:41:24.032111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:24.032001 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:24.561630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:24.561595 5049 generic.go:356] "Generic (PLEG): container finished" podID="3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852" containerID="86fbe0dd8554a16f1181c3da13107a78c63ab31740701606ab0c5255a76aeeba" exitCode=0 Oct 30 13:41:24.562042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:24.561684 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" event={"ID":"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852","Type":"ContainerDied","Data":"86fbe0dd8554a16f1181c3da13107a78c63ab31740701606ab0c5255a76aeeba"} Oct 30 13:41:25.904307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:25.904207 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:25.995145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:25.995110 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:26.107398 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.107361 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-xks-ca\") pod \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " Oct 30 13:41:26.107609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.107587 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-openshift-ca\") pod \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " Oct 30 13:41:26.107753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.107737 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-kube-api-access-2tnpj\" (UniqueName: \"kubernetes.io/projected/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-kube-api-access-2tnpj\") pod \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\" (UID: \"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852\") " Oct 30 13:41:26.107982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.107956 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-openshift-ca" pod "3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852" (UID: "3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:26.108530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.108503 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:26.109796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.109770 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-xks-ca" pod "3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852" (UID: "3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:26.109994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.109974 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-kube-api-access-2tnpj" pod "3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852" (UID: "3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852"). InnerVolumeSpecName "kube-api-access-2tnpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:26.210557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.210462 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:26.210557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.210495 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tnpj\" (UniqueName: \"kubernetes.io/projected/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852-kube-api-access-2tnpj\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:26.586144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.586119 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" Oct 30 13:41:26.586381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.586154 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6" event={"ID":"3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852","Type":"ContainerDied","Data":"d6ae28b6dc513b187a1b2e32c14abd382c7bf08110c606acaca64ca9410eb49f"} Oct 30 13:41:26.586381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:26.586213 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6ae28b6dc513b187a1b2e32c14abd382c7bf08110c606acaca64ca9410eb49f" Oct 30 13:41:26.678577 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:26.678539 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:41:27.885658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:27.885617 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:28.500777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:28.500742 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26"] Oct 30 13:41:28.526306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:28.526275 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n"] Oct 30 13:41:28.670353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:28.670294 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:28.828868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:28.828821 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:28.983466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:28.983424 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:29.908989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:29.908920 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:30.112196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:30.112161 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:30.261685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:30.261587 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:32.436338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:32.436280 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:33.501776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.501732 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" podUID="db8c604c-6b7e-42d8-8d5f-2d0104ba84f5" containerName="payload-processing" containerID="cri-o://eed40049badce1509ad5a031df0098011a42e2776013329bd9d56b7d239d2bf4" gracePeriod=25 Oct 30 13:41:33.527227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.527196 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" podUID="73d20ced-fe9a-4750-881d-835f9a94da0b" containerName="payload-pre-processing" containerID="cri-o://3caa1cc78b94febe5940a388c8827d52b74b7b604eb45060c5f388798c9809ff" gracePeriod=25 Oct 30 13:41:33.680136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.680085 5049 generic.go:356] "Generic (PLEG): container finished" podID="db8c604c-6b7e-42d8-8d5f-2d0104ba84f5" containerID="eed40049badce1509ad5a031df0098011a42e2776013329bd9d56b7d239d2bf4" exitCode=0 Oct 30 13:41:33.680354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.680225 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" event={"ID":"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5","Type":"ContainerDied","Data":"eed40049badce1509ad5a031df0098011a42e2776013329bd9d56b7d239d2bf4"} Oct 30 13:41:33.682219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.682187 5049 generic.go:356] "Generic (PLEG): container finished" podID="73d20ced-fe9a-4750-881d-835f9a94da0b" containerID="3caa1cc78b94febe5940a388c8827d52b74b7b604eb45060c5f388798c9809ff" exitCode=0 Oct 30 13:41:33.682430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.682415 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" event={"ID":"73d20ced-fe9a-4750-881d-835f9a94da0b","Type":"ContainerDied","Data":"3caa1cc78b94febe5940a388c8827d52b74b7b604eb45060c5f388798c9809ff"} Oct 30 13:41:33.820241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.820207 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:41:33.846673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.846639 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-kube-api-access-vw9ks\" (UniqueName: \"kubernetes.io/projected/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-kube-api-access-vw9ks\") pod \"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5\" (UID: \"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5\") " Oct 30 13:41:33.846906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.846885 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-plugins-config-volume\") pod \"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5\" (UID: \"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5\") " Oct 30 13:41:33.847378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.847311 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-plugins-config-volume" pod "db8c604c-6b7e-42d8-8d5f-2d0104ba84f5" (UID: "db8c604c-6b7e-42d8-8d5f-2d0104ba84f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:33.847663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.847641 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:33.848876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.848853 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-kube-api-access-vw9ks" pod "db8c604c-6b7e-42d8-8d5f-2d0104ba84f5" (UID: "db8c604c-6b7e-42d8-8d5f-2d0104ba84f5"). InnerVolumeSpecName "kube-api-access-vw9ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:33.882313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.882280 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:41:33.948419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.948379 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d20ced-fe9a-4750-881d-835f9a94da0b-kube-api-access-7pttv\" (UniqueName: \"kubernetes.io/projected/73d20ced-fe9a-4750-881d-835f9a94da0b-kube-api-access-7pttv\") pod \"73d20ced-fe9a-4750-881d-835f9a94da0b\" (UID: \"73d20ced-fe9a-4750-881d-835f9a94da0b\") " Oct 30 13:41:33.948667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.948647 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d20ced-fe9a-4750-881d-835f9a94da0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d20ced-fe9a-4750-881d-835f9a94da0b-plugins-config-volume\") pod \"73d20ced-fe9a-4750-881d-835f9a94da0b\" (UID: \"73d20ced-fe9a-4750-881d-835f9a94da0b\") " Oct 30 13:41:33.949600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.949569 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw9ks\" (UniqueName: \"kubernetes.io/projected/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5-kube-api-access-vw9ks\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:33.949850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.949825 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d20ced-fe9a-4750-881d-835f9a94da0b-plugins-config-volume" pod "73d20ced-fe9a-4750-881d-835f9a94da0b" (UID: "73d20ced-fe9a-4750-881d-835f9a94da0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:33.951801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:33.951770 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d20ced-fe9a-4750-881d-835f9a94da0b-kube-api-access-7pttv" pod "73d20ced-fe9a-4750-881d-835f9a94da0b" (UID: "73d20ced-fe9a-4750-881d-835f9a94da0b"). InnerVolumeSpecName "kube-api-access-7pttv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:34.011853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.011821 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:41:34.018215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.018186 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:41:34.051757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.051721 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pttv\" (UniqueName: \"kubernetes.io/projected/73d20ced-fe9a-4750-881d-835f9a94da0b-kube-api-access-7pttv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:34.051757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.051757 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d20ced-fe9a-4750-881d-835f9a94da0b-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:34.564513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.564481 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:41:34.564937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.564916 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:41:34.696359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.695141 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" Oct 30 13:41:34.696359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.695181 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26" event={"ID":"db8c604c-6b7e-42d8-8d5f-2d0104ba84f5","Type":"ContainerDied","Data":"2493980310cb4f4afb697d4d55b351e722075b2f491eda0b88a43bc68df023cc"} Oct 30 13:41:34.696359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.695253 5049 scope.go:122] "RemoveContainer" containerID="eed40049badce1509ad5a031df0098011a42e2776013329bd9d56b7d239d2bf4" Oct 30 13:41:34.697811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.697786 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" event={"ID":"73d20ced-fe9a-4750-881d-835f9a94da0b","Type":"ContainerDied","Data":"d6ca507c3d5f1fe1b623ce2d09f982493ef7458bd8e6b5060029218ff20b64e9"} Oct 30 13:41:34.697939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.697890 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n" Oct 30 13:41:34.710753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.710736 5049 scope.go:122] "RemoveContainer" containerID="3caa1cc78b94febe5940a388c8827d52b74b7b604eb45060c5f388798c9809ff" Oct 30 13:41:34.720944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.720917 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj"] Oct 30 13:41:34.721142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.721120 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerName="istio-proxy" containerID="cri-o://f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7" gracePeriod=30 Oct 30 13:41:34.725877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.725851 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt"] Oct 30 13:41:34.726200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.726058 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerName="istio-proxy" containerID="cri-o://8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5" gracePeriod=30 Oct 30 13:41:34.730650 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.730627 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc"] Oct 30 13:41:34.730801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.730777 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" podUID="88af2245-014a-4598-a442-7053d9d1e928" containerName="istio-proxy" containerID="cri-o://5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3" gracePeriod=30 Oct 30 13:41:34.746447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.746414 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26"] Oct 30 13:41:34.755397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.752672 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-57786bd588-2kx26"] Oct 30 13:41:34.836144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.836113 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n"] Oct 30 13:41:34.843441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.843404 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-78b767c9c9-cq86n"] Oct 30 13:41:34.899142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:34.899103 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:35.338112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:35.338079 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d20ced-fe9a-4750-881d-835f9a94da0b" path="/var/lib/kubelet/pods/73d20ced-fe9a-4750-881d-835f9a94da0b/volumes" Oct 30 13:41:35.338718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:35.338693 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8c604c-6b7e-42d8-8d5f-2d0104ba84f5" path="/var/lib/kubelet/pods/db8c604c-6b7e-42d8-8d5f-2d0104ba84f5/volumes" Oct 30 13:41:36.937544 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:36.937436 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:37.469966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:37.469929 5049 scope.go:122] "RemoveContainer" containerID="04497c44236e0af3f73c2cfa4469a5cf3accfe46f4fd3ea421e8c06a15bb7f21" Oct 30 13:41:37.481270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:37.481248 5049 scope.go:122] "RemoveContainer" containerID="1b046004257156ff6307e7b67a41aab9f5c809dab15fcc86d5b8646fe4961e6e" Oct 30 13:41:37.491673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:37.491648 5049 scope.go:122] "RemoveContainer" containerID="a05d9a68dea6d260835acf0dab56c6c1d123d7589decac4f616cfbf963dac186" Oct 30 13:41:37.543216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:37.543187 5049 scope.go:122] "RemoveContainer" containerID="3d349c9e7b75cc01055a223c40fe2953229c3bb3a1e655eb7eb9352951d7f420" Oct 30 13:41:37.905297 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:37.905261 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:38.079346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.079214 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:38.608220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.608185 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx"] Oct 30 13:41:38.609015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.608996 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" Oct 30 13:41:38.609076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609017 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" Oct 30 13:41:38.609076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609041 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" Oct 30 13:41:38.609076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609047 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" Oct 30 13:41:38.609076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609062 5049 cpu_manager.go:414] "containerMap: removing container" podUID="73d20ced-fe9a-4750-881d-835f9a94da0b" containerName="payload-pre-processing" Oct 30 13:41:38.609076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609068 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d20ced-fe9a-4750-881d-835f9a94da0b" containerName="payload-pre-processing" Oct 30 13:41:38.609076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609076 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852" containerName="revoke-keys" Oct 30 13:41:38.609270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609081 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852" containerName="revoke-keys" Oct 30 13:41:38.609270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609099 5049 cpu_manager.go:414] "containerMap: removing container" podUID="db8c604c-6b7e-42d8-8d5f-2d0104ba84f5" containerName="payload-processing" Oct 30 13:41:38.609270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609104 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8c604c-6b7e-42d8-8d5f-2d0104ba84f5" containerName="payload-processing" Oct 30 13:41:38.609270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609209 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d20ced-fe9a-4750-881d-835f9a94da0b" containerName="payload-pre-processing" Oct 30 13:41:38.609270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609216 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852" containerName="revoke-keys" Oct 30 13:41:38.609270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609224 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8c604c-6b7e-42d8-8d5f-2d0104ba84f5" containerName="payload-processing" Oct 30 13:41:38.609270 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.609234 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" Oct 30 13:41:38.616282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.616259 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.636825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.636796 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7"] Oct 30 13:41:38.638117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.637937 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c491c8a2-4e07-4851-aa5d-6a55dc247c59" containerName="istio-proxy" Oct 30 13:41:38.642416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.642395 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.645829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.645209 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:41:38.645829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.645217 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:41:38.660560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.651854 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx"] Oct 30 13:41:38.664635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.664608 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7"] Oct 30 13:41:38.718863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.717933 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v"] Oct 30 13:41:38.725886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.725853 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:38.740408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.740250 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v"] Oct 30 13:41:38.753191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.753160 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw6hw\" (UniqueName: \"kubernetes.io/projected/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-kube-api-access-tw6hw\") pod \"maas-api-revoke-sub-b34582ea-srgl7\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.753352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.753226 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjv6q\" (UniqueName: \"kubernetes.io/projected/dc9c017a-fd73-46aa-8974-186145c76297-kube-api-access-sjv6q\") pod \"maas-api-revoke-sub-56eadae9-mw8xx\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.753444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.753416 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b34582ea-srgl7\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.753493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.753480 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b34582ea-srgl7\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.753552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.753527 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56eadae9-mw8xx\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.753619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.753589 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56eadae9-mw8xx\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.855077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.855045 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b34582ea-srgl7\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.855264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.855093 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnlvm\" (UniqueName: \"kubernetes.io/projected/f165eb17-95a5-4da8-9627-b858f43b1100-kube-api-access-rnlvm\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:38.855264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.855118 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b34582ea-srgl7\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.855264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.855234 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56eadae9-mw8xx\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.855459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.855295 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56eadae9-mw8xx\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.855459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.855364 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:38.855575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.855459 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:38.855648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.855594 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw6hw\" (UniqueName: \"kubernetes.io/projected/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-kube-api-access-tw6hw\") pod \"maas-api-revoke-sub-b34582ea-srgl7\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.855708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.855686 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjv6q\" (UniqueName: \"kubernetes.io/projected/dc9c017a-fd73-46aa-8974-186145c76297-kube-api-access-sjv6q\") pod \"maas-api-revoke-sub-56eadae9-mw8xx\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.855991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.855963 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b34582ea-srgl7\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.856115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.856093 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56eadae9-mw8xx\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.857658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.857635 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b34582ea-srgl7\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.858932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.858880 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56eadae9-mw8xx\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.929362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.926199 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw6hw\" (UniqueName: \"kubernetes.io/projected/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-kube-api-access-tw6hw\") pod \"maas-api-revoke-sub-b34582ea-srgl7\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.933690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.932064 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjv6q\" (UniqueName: \"kubernetes.io/projected/dc9c017a-fd73-46aa-8974-186145c76297-kube-api-access-sjv6q\") pod \"maas-api-revoke-sub-56eadae9-mw8xx\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.933690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.932345 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:38.963970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.959242 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:38.963970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.959381 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:38.963970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.959607 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnlvm\" (UniqueName: \"kubernetes.io/projected/f165eb17-95a5-4da8-9627-b858f43b1100-kube-api-access-rnlvm\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:38.963970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.960754 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:38.984365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.974742 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:38.987382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.985250 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:38.992404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:38.989785 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnlvm\" (UniqueName: \"kubernetes.io/projected/f165eb17-95a5-4da8-9627-b858f43b1100-kube-api-access-rnlvm\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:39.050830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.050797 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:39.150635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.150296 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:39.341667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.341640 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx"] Oct 30 13:41:39.364656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.364624 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7"] Oct 30 13:41:39.380422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.380396 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj"] Oct 30 13:41:39.386651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.386629 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.405364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.405270 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj"] Oct 30 13:41:39.444429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.444400 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v"] Oct 30 13:41:39.473422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.473381 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.473658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.473637 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.473768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.473755 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz92d\" (UniqueName: \"kubernetes.io/projected/438941e6-a769-4861-b36d-2d9561a156a2-kube-api-access-bz92d\") pod \"maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.508536 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:41:39.508499 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf165eb17_95a5_4da8_9627_b858f43b1100.slice/crio-5f80be6d1bb64f92d4079854dc6cb869cc84addfd797b017b9791351c9a166cc WatchSource:0}: Error finding container 5f80be6d1bb64f92d4079854dc6cb869cc84addfd797b017b9791351c9a166cc: Status 404 returned error can't find the container with id 5f80be6d1bb64f92d4079854dc6cb869cc84addfd797b017b9791351c9a166cc Oct 30 13:41:39.575558 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.575525 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.575710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.575664 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz92d\" (UniqueName: \"kubernetes.io/projected/438941e6-a769-4861-b36d-2d9561a156a2-kube-api-access-bz92d\") pod \"maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.575814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.575798 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.576465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.576442 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.606787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.606750 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.606953 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.606925 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz92d\" (UniqueName: \"kubernetes.io/projected/438941e6-a769-4861-b36d-2d9561a156a2-kube-api-access-bz92d\") pod \"maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.721057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.721027 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:39.758388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.758266 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" event={"ID":"f165eb17-95a5-4da8-9627-b858f43b1100","Type":"ContainerStarted","Data":"5f80be6d1bb64f92d4079854dc6cb869cc84addfd797b017b9791351c9a166cc"} Oct 30 13:41:39.760704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.760671 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" event={"ID":"dc9c017a-fd73-46aa-8974-186145c76297","Type":"ContainerStarted","Data":"05b1b9137b3f77b87aaedd142d339c6fefb964e5f6efd08c933aff769bea8158"} Oct 30 13:41:39.760855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.760726 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" event={"ID":"dc9c017a-fd73-46aa-8974-186145c76297","Type":"ContainerStarted","Data":"ce0117c18c83841d7f2d7529db83db14bdb0faf74defd303e2d78bae2cafb999"} Oct 30 13:41:39.762088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.762062 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" event={"ID":"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633","Type":"ContainerStarted","Data":"3806c21163befb41647a68166ba9c21452bbba454fc711062851e0afc58a9252"} Oct 30 13:41:39.800970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:39.800907 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" podStartSLOduration=1.800888922 podStartE2EDuration="1.800888922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:39.796449297 +0000 UTC m=+3006.611643733" watchObservedRunningTime="2026-10-30 13:41:39.800888922 +0000 UTC m=+3006.616083347" Oct 30 13:41:40.138568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.133887 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj"] Oct 30 13:41:40.161554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.161201 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:41:40.163672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.163649 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:41:40.238664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.238582 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:41:40.290721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.290678 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe8ad4c-51e3-416f-a972-dfa870688e69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe8ad4c-51e3-416f-a972-dfa870688e69-istiod-ca-cert\") pod \"4fe8ad4c-51e3-416f-a972-dfa870688e69\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " Oct 30 13:41:40.291430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.291288 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe8ad4c-51e3-416f-a972-dfa870688e69-istiod-ca-cert" pod "4fe8ad4c-51e3-416f-a972-dfa870688e69" (UID: "4fe8ad4c-51e3-416f-a972-dfa870688e69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:40.291644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.291618 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-credential-socket\") pod \"4fe8ad4c-51e3-416f-a972-dfa870688e69\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " Oct 30 13:41:40.292123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.292007 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-credential-socket" pod "4fe8ad4c-51e3-416f-a972-dfa870688e69" (UID: "4fe8ad4c-51e3-416f-a972-dfa870688e69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.292575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.292556 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-envoy\") pod \"4fe8ad4c-51e3-416f-a972-dfa870688e69\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " Oct 30 13:41:40.292768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.292753 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-token\" (UniqueName: \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-token\") pod \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " Oct 30 13:41:40.293199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.293181 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-podinfo\") pod \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " Oct 30 13:41:40.293766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.293747 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-kube-api-access-q79tw\" (UniqueName: \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-kube-api-access-q79tw\") pod \"4fe8ad4c-51e3-416f-a972-dfa870688e69\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " Oct 30 13:41:40.294219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.294200 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-kube-api-access-mq2kr\" (UniqueName: \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-kube-api-access-mq2kr\") pod \"88af2245-014a-4598-a442-7053d9d1e928\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " Oct 30 13:41:40.294723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.294706 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istiod-ca-cert\") pod \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " Oct 30 13:41:40.295350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.295225 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istiod-ca-cert" pod "4503b1a1-fb99-4ea3-9c7d-e124fb841276" (UID: "4503b1a1-fb99-4ea3-9c7d-e124fb841276"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:40.298021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.297985 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-certs" pod "88af2245-014a-4598-a442-7053d9d1e928" (UID: "88af2245-014a-4598-a442-7053d9d1e928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.298276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.298235 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-certs\") pod \"88af2245-014a-4598-a442-7053d9d1e928\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " Oct 30 13:41:40.298445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.298351 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-token\" (UniqueName: \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-token\") pod \"4fe8ad4c-51e3-416f-a972-dfa870688e69\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.299925 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-credential-socket\") pod \"88af2245-014a-4598-a442-7053d9d1e928\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300020 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-socket\") pod \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300085 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-data\") pod \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300122 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-data\") pod \"4fe8ad4c-51e3-416f-a972-dfa870688e69\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300201 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88af2245-014a-4598-a442-7053d9d1e928-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88af2245-014a-4598-a442-7053d9d1e928-istio-podinfo\") pod \"88af2245-014a-4598-a442-7053d9d1e928\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300230 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-certs\") pod \"4fe8ad4c-51e3-416f-a972-dfa870688e69\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300262 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-socket\") pod \"88af2245-014a-4598-a442-7053d9d1e928\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300312 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-credential-socket\") pod \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300418 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-envoy\") pod \"88af2245-014a-4598-a442-7053d9d1e928\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300444 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-istio-token\" (UniqueName: \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-istio-token\") pod \"88af2245-014a-4598-a442-7053d9d1e928\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300537 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-podinfo\") pod \"4fe8ad4c-51e3-416f-a972-dfa870688e69\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300570 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-envoy\") pod \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300626 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-socket\") pod \"4fe8ad4c-51e3-416f-a972-dfa870688e69\" (UID: \"4fe8ad4c-51e3-416f-a972-dfa870688e69\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300656 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-kube-api-access-98wmr\" (UniqueName: \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-kube-api-access-98wmr\") pod \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300714 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-certs\") pod \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\" (UID: \"4503b1a1-fb99-4ea3-9c7d-e124fb841276\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300743 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-data\") pod \"88af2245-014a-4598-a442-7053d9d1e928\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " Oct 30 13:41:40.301487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.300799 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88af2245-014a-4598-a442-7053d9d1e928-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88af2245-014a-4598-a442-7053d9d1e928-istiod-ca-cert\") pod \"88af2245-014a-4598-a442-7053d9d1e928\" (UID: \"88af2245-014a-4598-a442-7053d9d1e928\") " Oct 30 13:41:40.302503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.301678 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-credential-socket" pod "4503b1a1-fb99-4ea3-9c7d-e124fb841276" (UID: "4503b1a1-fb99-4ea3-9c7d-e124fb841276"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.302503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.301853 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-credential-socket" pod "88af2245-014a-4598-a442-7053d9d1e928" (UID: "88af2245-014a-4598-a442-7053d9d1e928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.302503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.302011 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-socket" pod "4503b1a1-fb99-4ea3-9c7d-e124fb841276" (UID: "4503b1a1-fb99-4ea3-9c7d-e124fb841276"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.302503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.302167 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-data" pod "4503b1a1-fb99-4ea3-9c7d-e124fb841276" (UID: "4503b1a1-fb99-4ea3-9c7d-e124fb841276"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.302503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.302378 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-data" pod "4fe8ad4c-51e3-416f-a972-dfa870688e69" (UID: "4fe8ad4c-51e3-416f-a972-dfa870688e69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.303683 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe8ad4c-51e3-416f-a972-dfa870688e69-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.303710 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.303727 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.303741 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.303753 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.303767 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.303780 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.303792 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.303805 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.304223 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-certs" pod "4503b1a1-fb99-4ea3-9c7d-e124fb841276" (UID: "4503b1a1-fb99-4ea3-9c7d-e124fb841276"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.304658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.304397 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-data" pod "88af2245-014a-4598-a442-7053d9d1e928" (UID: "88af2245-014a-4598-a442-7053d9d1e928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.305601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.305005 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-socket" pod "4fe8ad4c-51e3-416f-a972-dfa870688e69" (UID: "4fe8ad4c-51e3-416f-a972-dfa870688e69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.305601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.305203 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-certs" pod "4fe8ad4c-51e3-416f-a972-dfa870688e69" (UID: "4fe8ad4c-51e3-416f-a972-dfa870688e69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.308105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.307465 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88af2245-014a-4598-a442-7053d9d1e928-istiod-ca-cert" pod "88af2245-014a-4598-a442-7053d9d1e928" (UID: "88af2245-014a-4598-a442-7053d9d1e928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:40.308105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.307559 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-kube-api-access-98wmr" pod "4503b1a1-fb99-4ea3-9c7d-e124fb841276" (UID: "4503b1a1-fb99-4ea3-9c7d-e124fb841276"). InnerVolumeSpecName "kube-api-access-98wmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:40.312043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.311750 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-socket" pod "88af2245-014a-4598-a442-7053d9d1e928" (UID: "88af2245-014a-4598-a442-7053d9d1e928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.313308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.312462 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88af2245-014a-4598-a442-7053d9d1e928-istio-podinfo" pod "88af2245-014a-4598-a442-7053d9d1e928" (UID: "88af2245-014a-4598-a442-7053d9d1e928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:40.313830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.313788 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-envoy" pod "4503b1a1-fb99-4ea3-9c7d-e124fb841276" (UID: "4503b1a1-fb99-4ea3-9c7d-e124fb841276"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.314530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.314484 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-podinfo" pod "4503b1a1-fb99-4ea3-9c7d-e124fb841276" (UID: "4503b1a1-fb99-4ea3-9c7d-e124fb841276"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:40.325547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.325525 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-envoy" pod "4fe8ad4c-51e3-416f-a972-dfa870688e69" (UID: "4fe8ad4c-51e3-416f-a972-dfa870688e69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.408540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.408508 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.408751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.408738 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.409145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.408936 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88af2245-014a-4598-a442-7053d9d1e928-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.411516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.411497 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.411647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.411633 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.411752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.411739 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88af2245-014a-4598-a442-7053d9d1e928-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.411849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.411838 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.411930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.411920 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.412012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.412000 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.412100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.412087 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe8ad4c-51e3-416f-a972-dfa870688e69-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.412198 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.412183 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98wmr\" (UniqueName: \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-kube-api-access-98wmr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.413493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.413465 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-token" pod "4503b1a1-fb99-4ea3-9c7d-e124fb841276" (UID: "4503b1a1-fb99-4ea3-9c7d-e124fb841276"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:40.413981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.413945 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-istio-token" pod "88af2245-014a-4598-a442-7053d9d1e928" (UID: "88af2245-014a-4598-a442-7053d9d1e928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:40.415164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.415132 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-kube-api-access-mq2kr" pod "88af2245-014a-4598-a442-7053d9d1e928" (UID: "88af2245-014a-4598-a442-7053d9d1e928"). InnerVolumeSpecName "kube-api-access-mq2kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:40.418243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.418217 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-envoy" pod "88af2245-014a-4598-a442-7053d9d1e928" (UID: "88af2245-014a-4598-a442-7053d9d1e928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:40.423787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.423755 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-kube-api-access-q79tw" pod "4fe8ad4c-51e3-416f-a972-dfa870688e69" (UID: "4fe8ad4c-51e3-416f-a972-dfa870688e69"). InnerVolumeSpecName "kube-api-access-q79tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:40.423787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.423754 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-podinfo" pod "4fe8ad4c-51e3-416f-a972-dfa870688e69" (UID: "4fe8ad4c-51e3-416f-a972-dfa870688e69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:40.423907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.423811 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-token" pod "4fe8ad4c-51e3-416f-a972-dfa870688e69" (UID: "4fe8ad4c-51e3-416f-a972-dfa870688e69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:40.513808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.513726 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88af2245-014a-4598-a442-7053d9d1e928-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.513808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.513762 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.513808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.513772 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.513808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.513781 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4503b1a1-fb99-4ea3-9c7d-e124fb841276-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.513808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.513790 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q79tw\" (UniqueName: \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-kube-api-access-q79tw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.513808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.513802 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq2kr\" (UniqueName: \"kubernetes.io/projected/88af2245-014a-4598-a442-7053d9d1e928-kube-api-access-mq2kr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.514116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.513815 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe8ad4c-51e3-416f-a972-dfa870688e69-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:40.773116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.773036 5049 generic.go:356] "Generic (PLEG): container finished" podID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerID="8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5" exitCode=0 Oct 30 13:41:40.773116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.773091 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" Oct 30 13:41:40.773358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.773124 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" event={"ID":"4fe8ad4c-51e3-416f-a972-dfa870688e69","Type":"ContainerDied","Data":"8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5"} Oct 30 13:41:40.773358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.773182 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt" event={"ID":"4fe8ad4c-51e3-416f-a972-dfa870688e69","Type":"ContainerDied","Data":"8d6a85442eeee22bcbf11f4d4f6021357e6284d947ad10cd8ca4219b8a60bca9"} Oct 30 13:41:40.773358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.773212 5049 scope.go:122] "RemoveContainer" containerID="8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5" Oct 30 13:41:40.774986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.774962 5049 generic.go:356] "Generic (PLEG): container finished" podID="88af2245-014a-4598-a442-7053d9d1e928" containerID="5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3" exitCode=0 Oct 30 13:41:40.775114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.775012 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" Oct 30 13:41:40.775114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.775051 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" event={"ID":"88af2245-014a-4598-a442-7053d9d1e928","Type":"ContainerDied","Data":"5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3"} Oct 30 13:41:40.775738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.775111 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc" event={"ID":"88af2245-014a-4598-a442-7053d9d1e928","Type":"ContainerDied","Data":"70974189a678c2c31355a80814abcc9ae62acdc2883dc3fe7fdfd278986bc98f"} Oct 30 13:41:40.776958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.776924 5049 generic.go:356] "Generic (PLEG): container finished" podID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerID="f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7" exitCode=0 Oct 30 13:41:40.777102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.776972 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" Oct 30 13:41:40.777102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.777023 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" event={"ID":"4503b1a1-fb99-4ea3-9c7d-e124fb841276","Type":"ContainerDied","Data":"f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7"} Oct 30 13:41:40.777102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.777064 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj" event={"ID":"4503b1a1-fb99-4ea3-9c7d-e124fb841276","Type":"ContainerDied","Data":"02f36e0faf50e00158f880cddaf83d235e16cc5e38952baa0ce608c5e779798c"} Oct 30 13:41:40.788498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.785932 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" event={"ID":"f165eb17-95a5-4da8-9627-b858f43b1100","Type":"ContainerStarted","Data":"509aef79f1a6ad1f10d7aa7bc3e25a966cb6c996d69ba0214c8570e8c7bdf8e7"} Oct 30 13:41:40.788760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.788735 5049 generic.go:356] "Generic (PLEG): container finished" podID="dc9c017a-fd73-46aa-8974-186145c76297" containerID="05b1b9137b3f77b87aaedd142d339c6fefb964e5f6efd08c933aff769bea8158" exitCode=0 Oct 30 13:41:40.788932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.788811 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" event={"ID":"dc9c017a-fd73-46aa-8974-186145c76297","Type":"ContainerDied","Data":"05b1b9137b3f77b87aaedd142d339c6fefb964e5f6efd08c933aff769bea8158"} Oct 30 13:41:40.791274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.791195 5049 generic.go:356] "Generic (PLEG): container finished" podID="5f2c343d-fa00-4a14-bb51-5eaf1ef0d633" containerID="0910571e1f3a84d0756ef582381d2a89d3b744b432c849960e1a1510a03e9dcb" exitCode=0 Oct 30 13:41:40.791274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.791269 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" event={"ID":"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633","Type":"ContainerDied","Data":"0910571e1f3a84d0756ef582381d2a89d3b744b432c849960e1a1510a03e9dcb"} Oct 30 13:41:40.791496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.791464 5049 scope.go:122] "RemoveContainer" containerID="8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5" Oct 30 13:41:40.791853 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:40.791828 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5\": container with ID starting with 8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5 not found: ID does not exist" containerID="8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5" Oct 30 13:41:40.791948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.791861 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5"} err="failed to get container status \"8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5\": rpc error: code = NotFound desc = could not find container \"8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5\": container with ID starting with 8a311848553f03d40e337f07be016b65ad794abf0ea17f9a7ab52112cdd727d5 not found: ID does not exist" Oct 30 13:41:40.791948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.791880 5049 scope.go:122] "RemoveContainer" containerID="5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3" Oct 30 13:41:40.793044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.793013 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" event={"ID":"438941e6-a769-4861-b36d-2d9561a156a2","Type":"ContainerStarted","Data":"5abbf456f2c7464e0e987b560aaf95668b9abe8412583fe7a6dd6c47a01e93ab"} Oct 30 13:41:40.793131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.793058 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" event={"ID":"438941e6-a769-4861-b36d-2d9561a156a2","Type":"ContainerStarted","Data":"e47eec8d63bf1afb11a6463879729bf894dcba58ec127e343812bac708b7162e"} Oct 30 13:41:40.805021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.805002 5049 scope.go:122] "RemoveContainer" containerID="5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3" Oct 30 13:41:40.805281 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:40.805265 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3\": container with ID starting with 5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3 not found: ID does not exist" containerID="5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3" Oct 30 13:41:40.805352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.805288 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3"} err="failed to get container status \"5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3\": rpc error: code = NotFound desc = could not find container \"5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3\": container with ID starting with 5fdd4a26f7210abc5cb4310f9fbef734f35224a897d2e88b8157ab7bf15d4de3 not found: ID does not exist" Oct 30 13:41:40.805352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.805305 5049 scope.go:122] "RemoveContainer" containerID="f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7" Oct 30 13:41:40.816301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.816283 5049 scope.go:122] "RemoveContainer" containerID="f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7" Oct 30 13:41:40.816599 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:40.816578 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7\": container with ID starting with f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7 not found: ID does not exist" containerID="f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7" Oct 30 13:41:40.816656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.816604 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7"} err="failed to get container status \"f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7\": rpc error: code = NotFound desc = could not find container \"f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7\": container with ID starting with f44335013aee9cf4f2f7a18e1bf52f03fddb3a555752457d35323d20850081d7 not found: ID does not exist" Oct 30 13:41:40.851965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.851920 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" podStartSLOduration=2.851907044 podStartE2EDuration="2.851907044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:40.848381964 +0000 UTC m=+3007.663576388" watchObservedRunningTime="2026-10-30 13:41:40.851907044 +0000 UTC m=+3007.667101466" Oct 30 13:41:40.885886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.885826 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" podStartSLOduration=1.88581258 podStartE2EDuration="1.88581258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:40.8843994 +0000 UTC m=+3007.699593822" watchObservedRunningTime="2026-10-30 13:41:40.88581258 +0000 UTC m=+3007.701007004" Oct 30 13:41:40.920432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.920406 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj"] Oct 30 13:41:40.929290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.929247 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-s4dtj"] Oct 30 13:41:40.967055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.967017 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc"] Oct 30 13:41:40.968096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:40.968073 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmkqc"] Oct 30 13:41:41.067803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:41.067770 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt"] Oct 30 13:41:41.072047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:41.072021 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vrnjt"] Oct 30 13:41:41.229208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:41.229174 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:41.340371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:41.340275 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" path="/var/lib/kubelet/pods/4503b1a1-fb99-4ea3-9c7d-e124fb841276/volumes" Oct 30 13:41:41.340972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:41.340945 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" path="/var/lib/kubelet/pods/4fe8ad4c-51e3-416f-a972-dfa870688e69/volumes" Oct 30 13:41:41.341540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:41.341521 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88af2245-014a-4598-a442-7053d9d1e928" path="/var/lib/kubelet/pods/88af2245-014a-4598-a442-7053d9d1e928/volumes" Oct 30 13:41:41.804582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:41.804552 5049 generic.go:356] "Generic (PLEG): container finished" podID="438941e6-a769-4861-b36d-2d9561a156a2" containerID="5abbf456f2c7464e0e987b560aaf95668b9abe8412583fe7a6dd6c47a01e93ab" exitCode=0 Oct 30 13:41:41.804777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:41.804635 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" event={"ID":"438941e6-a769-4861-b36d-2d9561a156a2","Type":"ContainerDied","Data":"5abbf456f2c7464e0e987b560aaf95668b9abe8412583fe7a6dd6c47a01e93ab"} Oct 30 13:41:41.809543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:41.809513 5049 generic.go:356] "Generic (PLEG): container finished" podID="f165eb17-95a5-4da8-9627-b858f43b1100" containerID="509aef79f1a6ad1f10d7aa7bc3e25a966cb6c996d69ba0214c8570e8c7bdf8e7" exitCode=0 Oct 30 13:41:41.809654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:41.809590 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" event={"ID":"f165eb17-95a5-4da8-9627-b858f43b1100","Type":"ContainerDied","Data":"509aef79f1a6ad1f10d7aa7bc3e25a966cb6c996d69ba0214c8570e8c7bdf8e7"} Oct 30 13:41:42.015004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.014661 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:42.055376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.055288 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:42.139171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.139137 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9c017a-fd73-46aa-8974-186145c76297-kube-api-access-sjv6q\" (UniqueName: \"kubernetes.io/projected/dc9c017a-fd73-46aa-8974-186145c76297-kube-api-access-sjv6q\") pod \"dc9c017a-fd73-46aa-8974-186145c76297\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " Oct 30 13:41:42.139373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.139214 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-kube-api-access-tw6hw\" (UniqueName: \"kubernetes.io/projected/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-kube-api-access-tw6hw\") pod \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " Oct 30 13:41:42.139373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.139277 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-openshift-ca\") pod \"dc9c017a-fd73-46aa-8974-186145c76297\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " Oct 30 13:41:42.139373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.139340 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-xks-ca\") pod \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " Oct 30 13:41:42.139606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.139401 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-openshift-ca\") pod \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\" (UID: \"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633\") " Oct 30 13:41:42.139679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.139660 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-xks-ca\") pod \"dc9c017a-fd73-46aa-8974-186145c76297\" (UID: \"dc9c017a-fd73-46aa-8974-186145c76297\") " Oct 30 13:41:42.141134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.141087 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-openshift-ca" pod "5f2c343d-fa00-4a14-bb51-5eaf1ef0d633" (UID: "5f2c343d-fa00-4a14-bb51-5eaf1ef0d633"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:42.141613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.141589 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-openshift-ca" pod "dc9c017a-fd73-46aa-8974-186145c76297" (UID: "dc9c017a-fd73-46aa-8974-186145c76297"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:42.142982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.142957 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-xks-ca" pod "dc9c017a-fd73-46aa-8974-186145c76297" (UID: "dc9c017a-fd73-46aa-8974-186145c76297"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:42.144726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.144699 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9c017a-fd73-46aa-8974-186145c76297-kube-api-access-sjv6q" pod "dc9c017a-fd73-46aa-8974-186145c76297" (UID: "dc9c017a-fd73-46aa-8974-186145c76297"). InnerVolumeSpecName "kube-api-access-sjv6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:42.145013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.144983 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-xks-ca" pod "5f2c343d-fa00-4a14-bb51-5eaf1ef0d633" (UID: "5f2c343d-fa00-4a14-bb51-5eaf1ef0d633"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:42.146223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.146198 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-kube-api-access-tw6hw" pod "5f2c343d-fa00-4a14-bb51-5eaf1ef0d633" (UID: "5f2c343d-fa00-4a14-bb51-5eaf1ef0d633"). InnerVolumeSpecName "kube-api-access-tw6hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:42.241450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.241415 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjv6q\" (UniqueName: \"kubernetes.io/projected/dc9c017a-fd73-46aa-8974-186145c76297-kube-api-access-sjv6q\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.241450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.241447 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw6hw\" (UniqueName: \"kubernetes.io/projected/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-kube-api-access-tw6hw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.241890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.241460 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.241890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.241475 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.241890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.241487 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.241890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.241500 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9c017a-fd73-46aa-8974-186145c76297-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.577830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.577794 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:42.767512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.767462 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88"] Oct 30 13:41:42.768858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.768829 5049 cpu_manager.go:414] "containerMap: removing container" podUID="dc9c017a-fd73-46aa-8974-186145c76297" containerName="revoke-keys" Oct 30 13:41:42.768858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.768857 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9c017a-fd73-46aa-8974-186145c76297" containerName="revoke-keys" Oct 30 13:41:42.769038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.768874 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerName="istio-proxy" Oct 30 13:41:42.769038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.768882 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerName="istio-proxy" Oct 30 13:41:42.769038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.768914 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5f2c343d-fa00-4a14-bb51-5eaf1ef0d633" containerName="revoke-keys" Oct 30 13:41:42.769038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.768922 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2c343d-fa00-4a14-bb51-5eaf1ef0d633" containerName="revoke-keys" Oct 30 13:41:42.769038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.768933 5049 cpu_manager.go:414] "containerMap: removing container" podUID="88af2245-014a-4598-a442-7053d9d1e928" containerName="istio-proxy" Oct 30 13:41:42.769038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.768940 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88af2245-014a-4598-a442-7053d9d1e928" containerName="istio-proxy" Oct 30 13:41:42.769038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.768958 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerName="istio-proxy" Oct 30 13:41:42.769038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.768965 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerName="istio-proxy" Oct 30 13:41:42.769413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.769151 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9c017a-fd73-46aa-8974-186145c76297" containerName="revoke-keys" Oct 30 13:41:42.769413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.769165 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4503b1a1-fb99-4ea3-9c7d-e124fb841276" containerName="istio-proxy" Oct 30 13:41:42.769413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.769178 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2c343d-fa00-4a14-bb51-5eaf1ef0d633" containerName="revoke-keys" Oct 30 13:41:42.769413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.769189 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="88af2245-014a-4598-a442-7053d9d1e928" containerName="istio-proxy" Oct 30 13:41:42.769413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.769200 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe8ad4c-51e3-416f-a972-dfa870688e69" containerName="istio-proxy" Oct 30 13:41:42.773600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.773575 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:42.820845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.820798 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" event={"ID":"dc9c017a-fd73-46aa-8974-186145c76297","Type":"ContainerDied","Data":"ce0117c18c83841d7f2d7529db83db14bdb0faf74defd303e2d78bae2cafb999"} Oct 30 13:41:42.821015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.820870 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce0117c18c83841d7f2d7529db83db14bdb0faf74defd303e2d78bae2cafb999" Oct 30 13:41:42.821015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.820816 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx" Oct 30 13:41:42.822622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.822587 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" event={"ID":"5f2c343d-fa00-4a14-bb51-5eaf1ef0d633","Type":"ContainerDied","Data":"3806c21163befb41647a68166ba9c21452bbba454fc711062851e0afc58a9252"} Oct 30 13:41:42.822754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.822657 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3806c21163befb41647a68166ba9c21452bbba454fc711062851e0afc58a9252" Oct 30 13:41:42.823023 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.822857 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7" Oct 30 13:41:42.831727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.831644 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88"] Oct 30 13:41:42.950763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.950540 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:42.950763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.950725 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:42.951021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.950839 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:42.951021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.950904 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzzrc\" (UniqueName: \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-kube-api-access-mzzrc\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:42.951021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.950938 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:42.951021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.950976 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:42.951260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.951021 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:42.951479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.951353 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:42.951556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:42.951476 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.048881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.048851 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:43.053282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.053253 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.053444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.053350 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzzrc\" (UniqueName: \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-kube-api-access-mzzrc\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.053444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.053388 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.053444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.053427 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.053617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.053471 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.053617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.053604 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.053729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.053645 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.053729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.053687 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.053850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.053791 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.054049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.054020 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.054332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.054293 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.054817 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.054774 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.055062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.055034 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.055146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.055095 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.057015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.056992 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.057234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.057210 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.075815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.075780 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.081494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.081466 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzzrc\" (UniqueName: \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-kube-api-access-mzzrc\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.089314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.089241 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:43.112692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.112665 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:43.162474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.156897 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-xks-ca\") pod \"438941e6-a769-4861-b36d-2d9561a156a2\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " Oct 30 13:41:43.162474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.157076 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438941e6-a769-4861-b36d-2d9561a156a2-kube-api-access-bz92d\" (UniqueName: \"kubernetes.io/projected/438941e6-a769-4861-b36d-2d9561a156a2-kube-api-access-bz92d\") pod \"438941e6-a769-4861-b36d-2d9561a156a2\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " Oct 30 13:41:43.162474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.157186 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-openshift-ca\") pod \"438941e6-a769-4861-b36d-2d9561a156a2\" (UID: \"438941e6-a769-4861-b36d-2d9561a156a2\") " Oct 30 13:41:43.162474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.158542 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-openshift-ca" pod "438941e6-a769-4861-b36d-2d9561a156a2" (UID: "438941e6-a769-4861-b36d-2d9561a156a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:43.162474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.160683 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-xks-ca" pod "438941e6-a769-4861-b36d-2d9561a156a2" (UID: "438941e6-a769-4861-b36d-2d9561a156a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:43.162474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.160939 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438941e6-a769-4861-b36d-2d9561a156a2-kube-api-access-bz92d" pod "438941e6-a769-4861-b36d-2d9561a156a2" (UID: "438941e6-a769-4861-b36d-2d9561a156a2"). InnerVolumeSpecName "kube-api-access-bz92d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:43.259412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.259370 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-xks-ca\") pod \"f165eb17-95a5-4da8-9627-b858f43b1100\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " Oct 30 13:41:43.259941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.259536 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f165eb17-95a5-4da8-9627-b858f43b1100-kube-api-access-rnlvm\" (UniqueName: \"kubernetes.io/projected/f165eb17-95a5-4da8-9627-b858f43b1100-kube-api-access-rnlvm\") pod \"f165eb17-95a5-4da8-9627-b858f43b1100\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " Oct 30 13:41:43.259941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.259710 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-openshift-ca\") pod \"f165eb17-95a5-4da8-9627-b858f43b1100\" (UID: \"f165eb17-95a5-4da8-9627-b858f43b1100\") " Oct 30 13:41:43.260248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.260220 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-openshift-ca" pod "f165eb17-95a5-4da8-9627-b858f43b1100" (UID: "f165eb17-95a5-4da8-9627-b858f43b1100"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:43.260491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.260472 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:43.260598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.260495 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438941e6-a769-4861-b36d-2d9561a156a2-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:43.260598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.260509 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz92d\" (UniqueName: \"kubernetes.io/projected/438941e6-a769-4861-b36d-2d9561a156a2-kube-api-access-bz92d\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:43.260598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.260528 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:43.262535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.262505 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f165eb17-95a5-4da8-9627-b858f43b1100-kube-api-access-rnlvm" pod "f165eb17-95a5-4da8-9627-b858f43b1100" (UID: "f165eb17-95a5-4da8-9627-b858f43b1100"). InnerVolumeSpecName "kube-api-access-rnlvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:43.262724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.262700 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-xks-ca" pod "f165eb17-95a5-4da8-9627-b858f43b1100" (UID: "f165eb17-95a5-4da8-9627-b858f43b1100"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:43.361916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.361887 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f165eb17-95a5-4da8-9627-b858f43b1100-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:43.361916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.361917 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnlvm\" (UniqueName: \"kubernetes.io/projected/f165eb17-95a5-4da8-9627-b858f43b1100-kube-api-access-rnlvm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:43.443276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.443249 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88"] Oct 30 13:41:43.445014 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:41:43.444987 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bb0776_30fd_4a9d_8d81_b1a16ee10fb4.slice/crio-dc6cd477d8674a2e64c71ae780ba7a96e74a4f6dc6092c9f25ac8a5655a20b76 WatchSource:0}: Error finding container dc6cd477d8674a2e64c71ae780ba7a96e74a4f6dc6092c9f25ac8a5655a20b76: Status 404 returned error can't find the container with id dc6cd477d8674a2e64c71ae780ba7a96e74a4f6dc6092c9f25ac8a5655a20b76 Oct 30 13:41:43.448776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.448738 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:41:43.448854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.448831 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:41:43.448902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.448877 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:41:43.833891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.833860 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" Oct 30 13:41:43.833891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.833876 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v" event={"ID":"f165eb17-95a5-4da8-9627-b858f43b1100","Type":"ContainerDied","Data":"5f80be6d1bb64f92d4079854dc6cb869cc84addfd797b017b9791351c9a166cc"} Oct 30 13:41:43.834139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.833961 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f80be6d1bb64f92d4079854dc6cb869cc84addfd797b017b9791351c9a166cc" Oct 30 13:41:43.835906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.835729 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" event={"ID":"438941e6-a769-4861-b36d-2d9561a156a2","Type":"ContainerDied","Data":"e47eec8d63bf1afb11a6463879729bf894dcba58ec127e343812bac708b7162e"} Oct 30 13:41:43.835906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.835775 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj" Oct 30 13:41:43.835906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.835775 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e47eec8d63bf1afb11a6463879729bf894dcba58ec127e343812bac708b7162e" Oct 30 13:41:43.838766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.838740 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" event={"ID":"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4","Type":"ContainerStarted","Data":"229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01"} Oct 30 13:41:43.838883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.838786 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" event={"ID":"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4","Type":"ContainerStarted","Data":"dc6cd477d8674a2e64c71ae780ba7a96e74a4f6dc6092c9f25ac8a5655a20b76"} Oct 30 13:41:43.889427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.888076 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:43.914902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:43.914848 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" podStartSLOduration=1.914832241 podStartE2EDuration="1.914832241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:43.904753084 +0000 UTC m=+3010.719947550" watchObservedRunningTime="2026-10-30 13:41:43.914832241 +0000 UTC m=+3010.730026664" Oct 30 13:41:44.090204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:44.090081 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:44.092135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:44.092107 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 30 13:41:44.092268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:44.092163 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 30 13:41:45.089728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:45.089693 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 30 13:41:45.090178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:45.089759 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 30 13:41:46.051342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:46.048297 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:46.089682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:46.089647 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 30 13:41:46.089872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:46.089722 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 30 13:41:47.090168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:47.090131 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 30 13:41:47.090597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:47.090183 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 30 13:41:47.663747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:47.663710 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:47.820343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:47.819814 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:48.089669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:48.089629 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 30 13:41:48.089883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:48.089686 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 30 13:41:48.129859 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:48.129825 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:49.090592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:49.090555 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Oct 30 13:41:49.090787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:49.090608 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Oct 30 13:41:50.095039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:50.095008 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:50.095560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:50.095310 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:50.095953 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:50.095934 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:41:50.621381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:50.617891 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:52.407106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:52.407030 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:53.020527 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.020493 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj"] Oct 30 13:41:53.020704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.020684 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" podUID="9eaee4df-ebf8-4573-828d-a9acbc068eed" containerName="discovery" containerID="cri-o://0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426" gracePeriod=30 Oct 30 13:41:53.361926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.361895 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:41:53.446153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.446116 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaee4df-ebf8-4573-828d-a9acbc068eed-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaee4df-ebf8-4573-828d-a9acbc068eed-local-certs\") pod \"9eaee4df-ebf8-4573-828d-a9acbc068eed\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " Oct 30 13:41:53.447263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.447237 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-kube-api-access-pnpfr\" (UniqueName: \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-kube-api-access-pnpfr\") pod \"9eaee4df-ebf8-4573-828d-a9acbc068eed\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " Oct 30 13:41:53.448396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.448355 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-ca-configmap" pod "9eaee4df-ebf8-4573-828d-a9acbc068eed" (UID: "9eaee4df-ebf8-4573-828d-a9acbc068eed"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:53.448586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.448569 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-ca-configmap\") pod \"9eaee4df-ebf8-4573-828d-a9acbc068eed\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " Oct 30 13:41:53.448829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.448811 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-token\" (UniqueName: \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-token\") pod \"9eaee4df-ebf8-4573-828d-a9acbc068eed\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " Oct 30 13:41:53.449345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.449311 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-dns-cert\") pod \"9eaee4df-ebf8-4573-828d-a9acbc068eed\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " Oct 30 13:41:53.449539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.449519 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-cacerts\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-cacerts\") pod \"9eaee4df-ebf8-4573-828d-a9acbc068eed\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " Oct 30 13:41:53.449728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.449709 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-kubeconfig\") pod \"9eaee4df-ebf8-4573-828d-a9acbc068eed\" (UID: \"9eaee4df-ebf8-4573-828d-a9acbc068eed\") " Oct 30 13:41:53.450728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.450704 5049 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-ca-configmap\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:53.453966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.453930 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-dns-cert" pod "9eaee4df-ebf8-4573-828d-a9acbc068eed" (UID: "9eaee4df-ebf8-4573-828d-a9acbc068eed"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:53.454057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.454024 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-token" pod "9eaee4df-ebf8-4573-828d-a9acbc068eed" (UID: "9eaee4df-ebf8-4573-828d-a9acbc068eed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:53.454490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.454464 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaee4df-ebf8-4573-828d-a9acbc068eed-local-certs" pod "9eaee4df-ebf8-4573-828d-a9acbc068eed" (UID: "9eaee4df-ebf8-4573-828d-a9acbc068eed"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:53.457616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.457591 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-cacerts" pod "9eaee4df-ebf8-4573-828d-a9acbc068eed" (UID: "9eaee4df-ebf8-4573-828d-a9acbc068eed"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:53.457745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.457611 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-kubeconfig" pod "9eaee4df-ebf8-4573-828d-a9acbc068eed" (UID: "9eaee4df-ebf8-4573-828d-a9acbc068eed"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:53.457745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.457635 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-kube-api-access-pnpfr" pod "9eaee4df-ebf8-4573-828d-a9acbc068eed" (UID: "9eaee4df-ebf8-4573-828d-a9acbc068eed"). InnerVolumeSpecName "kube-api-access-pnpfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:53.552736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.552702 5049 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-csr-dns-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:53.552736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.552734 5049 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-cacerts\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:53.552736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.552745 5049 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-kubeconfig\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:53.552990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.552754 5049 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaee4df-ebf8-4573-828d-a9acbc068eed-local-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:53.552990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.552762 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnpfr\" (UniqueName: \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-kube-api-access-pnpfr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:53.552990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.552771 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaee4df-ebf8-4573-828d-a9acbc068eed-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:41:53.925872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.925831 5049 generic.go:356] "Generic (PLEG): container finished" podID="9eaee4df-ebf8-4573-828d-a9acbc068eed" containerID="0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426" exitCode=0 Oct 30 13:41:53.926110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.925894 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" Oct 30 13:41:53.926110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.926061 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" event={"ID":"9eaee4df-ebf8-4573-828d-a9acbc068eed","Type":"ContainerDied","Data":"0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426"} Oct 30 13:41:53.926110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.926104 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj" event={"ID":"9eaee4df-ebf8-4573-828d-a9acbc068eed","Type":"ContainerDied","Data":"0816f934da39293b5c6728820d168f3c8e467a11b1098f738ccb031790bf5f83"} Oct 30 13:41:53.926314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.926129 5049 scope.go:122] "RemoveContainer" containerID="0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426" Oct 30 13:41:53.939403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.939383 5049 scope.go:122] "RemoveContainer" containerID="0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426" Oct 30 13:41:53.939729 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:53.939702 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426\": container with ID starting with 0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426 not found: ID does not exist" containerID="0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426" Oct 30 13:41:53.939830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.939742 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426"} err="failed to get container status \"0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426\": rpc error: code = NotFound desc = could not find container \"0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426\": container with ID starting with 0b230929421553c95ab5e3adbff2deb73677eb5703da920b0d21f9487dabe426 not found: ID does not exist" Oct 30 13:41:53.991676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:53.991641 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj"] Oct 30 13:41:54.001578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:54.001535 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv4kj"] Oct 30 13:41:54.276737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:54.276453 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:55.336958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:55.336924 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eaee4df-ebf8-4573-828d-a9acbc068eed" path="/var/lib/kubelet/pods/9eaee4df-ebf8-4573-828d-a9acbc068eed/volumes" Oct 30 13:41:56.186461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:56.186431 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:56.678952 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:41:56.678914 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:41:58.225017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:58.224978 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:59.592239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:59.592197 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:59.764225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:59.763895 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:59.995421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:59.987687 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk"] Oct 30 13:42:00.000352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:41:59.996771 5049 cpu_manager.go:414] "containerMap: removing container" podUID="438941e6-a769-4861-b36d-2d9561a156a2" containerName="revoke-keys" Oct 30 13:42:00.000619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.000585 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="438941e6-a769-4861-b36d-2d9561a156a2" containerName="revoke-keys" Oct 30 13:42:00.008343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.000755 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f165eb17-95a5-4da8-9627-b858f43b1100" containerName="revoke-keys" Oct 30 13:42:00.008592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.008569 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f165eb17-95a5-4da8-9627-b858f43b1100" containerName="revoke-keys" Oct 30 13:42:00.008792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.008772 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9eaee4df-ebf8-4573-828d-a9acbc068eed" containerName="discovery" Oct 30 13:42:00.008918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.008903 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eaee4df-ebf8-4573-828d-a9acbc068eed" containerName="discovery" Oct 30 13:42:00.009307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.009286 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f165eb17-95a5-4da8-9627-b858f43b1100" containerName="revoke-keys" Oct 30 13:42:00.009475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.009462 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eaee4df-ebf8-4573-828d-a9acbc068eed" containerName="discovery" Oct 30 13:42:00.009579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.009567 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="438941e6-a769-4861-b36d-2d9561a156a2" containerName="revoke-keys" Oct 30 13:42:00.052512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.015412 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk"] Oct 30 13:42:00.052512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.015535 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:42:00.052512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.032247 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e3ebf5-dockercfg-bg99r\"" Oct 30 13:42:00.052512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.044589 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e3ebf5\"" Oct 30 13:42:00.098805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.098766 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk\" (UID: \"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:42:00.099075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.098861 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gz2\" (UniqueName: \"kubernetes.io/projected/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-kube-api-access-f2gz2\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk\" (UID: \"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:42:00.125342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.118621 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt"] Oct 30 13:42:00.135372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.133763 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:42:00.154341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.152957 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt"] Oct 30 13:42:00.200941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.200905 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gz2\" (UniqueName: \"kubernetes.io/projected/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-kube-api-access-f2gz2\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk\" (UID: \"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:42:00.201165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.201144 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwbc\" (UniqueName: \"kubernetes.io/projected/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-kube-api-access-nnwbc\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt\" (UID: \"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:42:00.201295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.201267 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt\" (UID: \"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:42:00.201395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.201377 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk\" (UID: \"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:42:00.202412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.202390 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk\" (UID: \"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:42:00.225499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.225472 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gz2\" (UniqueName: \"kubernetes.io/projected/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-kube-api-access-f2gz2\") pod \"payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk\" (UID: \"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:42:00.302332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.302296 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt\" (UID: \"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:42:00.302512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.302502 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwbc\" (UniqueName: \"kubernetes.io/projected/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-kube-api-access-nnwbc\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt\" (UID: \"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:42:00.303116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.303095 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt\" (UID: \"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:42:00.326885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.326857 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwbc\" (UniqueName: \"kubernetes.io/projected/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-kube-api-access-nnwbc\") pod \"payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt\" (UID: \"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:42:00.353981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.353950 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:42:00.459822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.459794 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:42:00.669048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.669021 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk"] Oct 30 13:42:00.670881 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:42:00.670853 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf4f011_ec24_40c7_a0ab_2bb59f48ad36.slice/crio-bbfb25959458369cebef840a86a6a31c7b6debca90cedf0a6d980ef018529e66 WatchSource:0}: Error finding container bbfb25959458369cebef840a86a6a31c7b6debca90cedf0a6d980ef018529e66: Status 404 returned error can't find the container with id bbfb25959458369cebef840a86a6a31c7b6debca90cedf0a6d980ef018529e66 Oct 30 13:42:00.824611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.824578 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt"] Oct 30 13:42:00.908240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:00.908207 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:01.015143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:01.015104 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" event={"ID":"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e","Type":"ContainerStarted","Data":"7e6073c6f385d6669e77512999d9f656d4406d3bae239b87005b970d01f6c1ad"} Oct 30 13:42:01.015382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:01.015166 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" event={"ID":"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e","Type":"ContainerStarted","Data":"b0cf265fa98aef7e3446173e0062edfde643668cd6952ae4ec5ed872d5a14bb2"} Oct 30 13:42:01.015382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:01.015217 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:42:01.017769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:01.017279 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" event={"ID":"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36","Type":"ContainerStarted","Data":"f66176d978888c8b1242d6c79f39bc8dbab7ae071696f22c8c252987868c2a14"} Oct 30 13:42:01.017769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:01.017340 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" event={"ID":"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36","Type":"ContainerStarted","Data":"bbfb25959458369cebef840a86a6a31c7b6debca90cedf0a6d980ef018529e66"} Oct 30 13:42:01.017769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:01.017399 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:42:01.058853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:01.058797 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" podStartSLOduration=1.058782774 podStartE2EDuration="1.058782774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:01.053593534 +0000 UTC m=+3027.868787961" watchObservedRunningTime="2026-10-30 13:42:01.058782774 +0000 UTC m=+3027.873977197" Oct 30 13:42:01.097571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:01.097516 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" podStartSLOduration=2.097502011 podStartE2EDuration="2.097502011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:01.092738559 +0000 UTC m=+3027.907932981" watchObservedRunningTime="2026-10-30 13:42:01.097502011 +0000 UTC m=+3027.912696433" Oct 30 13:42:02.979573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:02.979533 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5"] Oct 30 13:42:02.984225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:02.984199 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:02.997938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:02.997918 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:42:03.016346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.016306 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5"] Oct 30 13:42:03.065444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.065404 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b28ef90-558g5\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:03.065659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.065614 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdv5\" (UniqueName: \"kubernetes.io/projected/df406730-f0d4-4845-ada4-6d367fe059ce-kube-api-access-ctdv5\") pod \"maas-api-revoke-sub-1b28ef90-558g5\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:03.065769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.065751 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b28ef90-558g5\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:03.098465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.098435 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:42:03.166980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.166945 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b28ef90-558g5\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:03.167183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.167100 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdv5\" (UniqueName: \"kubernetes.io/projected/df406730-f0d4-4845-ada4-6d367fe059ce-kube-api-access-ctdv5\") pod \"maas-api-revoke-sub-1b28ef90-558g5\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:03.167255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.167195 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b28ef90-558g5\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:03.167872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.167846 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b28ef90-558g5\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:03.169509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.169489 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b28ef90-558g5\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:03.188649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.188629 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdv5\" (UniqueName: \"kubernetes.io/projected/df406730-f0d4-4845-ada4-6d367fe059ce-kube-api-access-ctdv5\") pod \"maas-api-revoke-sub-1b28ef90-558g5\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:03.299838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.299808 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:03.404172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.404132 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:03.617038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:03.617011 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5"] Oct 30 13:42:03.621024 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:42:03.620996 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf406730_f0d4_4845_ada4_6d367fe059ce.slice/crio-b6a4d6f3bc9d9cd153740d2b910a7897ed795916919d118d8c2b7259dbdc9236 WatchSource:0}: Error finding container b6a4d6f3bc9d9cd153740d2b910a7897ed795916919d118d8c2b7259dbdc9236: Status 404 returned error can't find the container with id b6a4d6f3bc9d9cd153740d2b910a7897ed795916919d118d8c2b7259dbdc9236 Oct 30 13:42:04.055878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:04.055837 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" event={"ID":"df406730-f0d4-4845-ada4-6d367fe059ce","Type":"ContainerStarted","Data":"27616f09e1b9fb7a08488b50460d67eb0755b5d2ad0fff83d3650f7bc186c303"} Oct 30 13:42:04.056286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:04.055905 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" event={"ID":"df406730-f0d4-4845-ada4-6d367fe059ce","Type":"ContainerStarted","Data":"b6a4d6f3bc9d9cd153740d2b910a7897ed795916919d118d8c2b7259dbdc9236"} Oct 30 13:42:04.093300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:04.093239 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" podStartSLOduration=2.093221966 podStartE2EDuration="2.093221966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:04.091274781 +0000 UTC m=+3030.906469205" watchObservedRunningTime="2026-10-30 13:42:04.093221966 +0000 UTC m=+3030.908416391" Oct 30 13:42:05.065932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:05.065897 5049 generic.go:356] "Generic (PLEG): container finished" podID="df406730-f0d4-4845-ada4-6d367fe059ce" containerID="27616f09e1b9fb7a08488b50460d67eb0755b5d2ad0fff83d3650f7bc186c303" exitCode=0 Oct 30 13:42:05.066357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:05.065980 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" event={"ID":"df406730-f0d4-4845-ada4-6d367fe059ce","Type":"ContainerDied","Data":"27616f09e1b9fb7a08488b50460d67eb0755b5d2ad0fff83d3650f7bc186c303"} Oct 30 13:42:05.276968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:05.275260 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:06.258878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:06.258851 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:06.312622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:06.312585 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-xks-ca\") pod \"df406730-f0d4-4845-ada4-6d367fe059ce\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " Oct 30 13:42:06.312824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:06.312657 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-openshift-ca\") pod \"df406730-f0d4-4845-ada4-6d367fe059ce\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " Oct 30 13:42:06.312933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:06.312917 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df406730-f0d4-4845-ada4-6d367fe059ce-kube-api-access-ctdv5\" (UniqueName: \"kubernetes.io/projected/df406730-f0d4-4845-ada4-6d367fe059ce-kube-api-access-ctdv5\") pod \"df406730-f0d4-4845-ada4-6d367fe059ce\" (UID: \"df406730-f0d4-4845-ada4-6d367fe059ce\") " Oct 30 13:42:06.313053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:06.313027 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-openshift-ca" pod "df406730-f0d4-4845-ada4-6d367fe059ce" (UID: "df406730-f0d4-4845-ada4-6d367fe059ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:06.313610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:06.313587 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:06.315719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:06.315663 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df406730-f0d4-4845-ada4-6d367fe059ce-kube-api-access-ctdv5" pod "df406730-f0d4-4845-ada4-6d367fe059ce" (UID: "df406730-f0d4-4845-ada4-6d367fe059ce"). InnerVolumeSpecName "kube-api-access-ctdv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:06.315964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:06.315936 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-xks-ca" pod "df406730-f0d4-4845-ada4-6d367fe059ce" (UID: "df406730-f0d4-4845-ada4-6d367fe059ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:06.416869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:06.416831 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdv5\" (UniqueName: \"kubernetes.io/projected/df406730-f0d4-4845-ada4-6d367fe059ce-kube-api-access-ctdv5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:06.417087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:06.417067 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df406730-f0d4-4845-ada4-6d367fe059ce-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:07.043407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:07.043368 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:07.084922 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:07.084891 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" Oct 30 13:42:07.085099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:07.084927 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5" event={"ID":"df406730-f0d4-4845-ada4-6d367fe059ce","Type":"ContainerDied","Data":"b6a4d6f3bc9d9cd153740d2b910a7897ed795916919d118d8c2b7259dbdc9236"} Oct 30 13:42:07.085099 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:07.085002 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6a4d6f3bc9d9cd153740d2b910a7897ed795916919d118d8c2b7259dbdc9236" Oct 30 13:42:08.923341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:08.923018 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:11.380686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:11.380626 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:12.029792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:12.029764 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:42:12.029970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:12.029840 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:42:13.230156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:13.230114 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:15.040251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:15.040215 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:16.417646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.417613 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:16.590382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.589945 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:16.817341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.812598 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq"] Oct 30 13:42:16.817341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.813857 5049 cpu_manager.go:414] "containerMap: removing container" podUID="df406730-f0d4-4845-ada4-6d367fe059ce" containerName="revoke-keys" Oct 30 13:42:16.817341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.813882 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df406730-f0d4-4845-ada4-6d367fe059ce" containerName="revoke-keys" Oct 30 13:42:16.817341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.814164 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="df406730-f0d4-4845-ada4-6d367fe059ce" containerName="revoke-keys" Oct 30 13:42:16.843346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.842603 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq"] Oct 30 13:42:16.843346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.842731 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:42:16.867840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.858602 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8ee35e\"" Oct 30 13:42:16.868289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.868227 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8ee35e-dockercfg-4gbnn\"" Oct 30 13:42:16.937009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.933104 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq\" (UID: \"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:42:16.937009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.933237 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvl2n\" (UniqueName: \"kubernetes.io/projected/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-kube-api-access-cvl2n\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq\" (UID: \"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:42:16.942173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.941493 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls"] Oct 30 13:42:16.951405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.951223 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:42:16.982937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:16.976940 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls"] Oct 30 13:42:17.034127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.034085 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvl2n\" (UniqueName: \"kubernetes.io/projected/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-kube-api-access-cvl2n\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq\" (UID: \"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:42:17.034372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.034140 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wff25\" (UniqueName: \"kubernetes.io/projected/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-kube-api-access-wff25\") pod \"payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls\" (UID: \"56ffe634-75ee-4ed1-abea-7dda3aad4a0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:42:17.034372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.034221 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls\" (UID: \"56ffe634-75ee-4ed1-abea-7dda3aad4a0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:42:17.034568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.034547 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq\" (UID: \"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:42:17.035223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.035204 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq\" (UID: \"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:42:17.065306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.065271 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvl2n\" (UniqueName: \"kubernetes.io/projected/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-kube-api-access-cvl2n\") pod \"payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq\" (UID: \"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:42:17.135572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.135486 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wff25\" (UniqueName: \"kubernetes.io/projected/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-kube-api-access-wff25\") pod \"payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls\" (UID: \"56ffe634-75ee-4ed1-abea-7dda3aad4a0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:42:17.135572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.135526 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls\" (UID: \"56ffe634-75ee-4ed1-abea-7dda3aad4a0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:42:17.136237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.136217 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls\" (UID: \"56ffe634-75ee-4ed1-abea-7dda3aad4a0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:42:17.157554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.157529 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wff25\" (UniqueName: \"kubernetes.io/projected/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-kube-api-access-wff25\") pod \"payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls\" (UID: \"56ffe634-75ee-4ed1-abea-7dda3aad4a0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:42:17.251925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.251892 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:42:17.303531 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.303496 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:42:17.601261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.601236 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq"] Oct 30 13:42:17.604294 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:42:17.604270 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5e4b9d_f3a9_40d8_9eac_30218c27dcfd.slice/crio-18d2ad9e13e2527e891af2a390286a6f021efe815d4da3ff9a07451e8327ba7f WatchSource:0}: Error finding container 18d2ad9e13e2527e891af2a390286a6f021efe815d4da3ff9a07451e8327ba7f: Status 404 returned error can't find the container with id 18d2ad9e13e2527e891af2a390286a6f021efe815d4da3ff9a07451e8327ba7f Oct 30 13:42:17.662253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.662223 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls"] Oct 30 13:42:17.662776 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:42:17.662746 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ffe634_75ee_4ed1_abea_7dda3aad4a0a.slice/crio-8f7e9dfb059367506666379d24bf2df49edc581ff93ef92929a1d867ec61473c WatchSource:0}: Error finding container 8f7e9dfb059367506666379d24bf2df49edc581ff93ef92929a1d867ec61473c: Status 404 returned error can't find the container with id 8f7e9dfb059367506666379d24bf2df49edc581ff93ef92929a1d867ec61473c Oct 30 13:42:17.968104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:17.968024 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:18.026435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.026401 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk"] Oct 30 13:42:18.032357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.032306 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.048150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.048126 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:42:18.048486 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.048464 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:42:18.051541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.051498 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx2wv\" (UniqueName: \"kubernetes.io/projected/0065d53d-a758-49dc-a2a5-9b50710eac2b-kube-api-access-qx2wv\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.056785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.054656 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk"] Oct 30 13:42:18.056785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.056677 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.056785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.056740 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.158443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.158403 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.158634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.158458 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.158634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.158599 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx2wv\" (UniqueName: \"kubernetes.io/projected/0065d53d-a758-49dc-a2a5-9b50710eac2b-kube-api-access-qx2wv\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.159156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.159132 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.160961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.160938 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.179877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.179851 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx2wv\" (UniqueName: \"kubernetes.io/projected/0065d53d-a758-49dc-a2a5-9b50710eac2b-kube-api-access-qx2wv\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.199095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.199056 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" event={"ID":"56ffe634-75ee-4ed1-abea-7dda3aad4a0a","Type":"ContainerStarted","Data":"1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d"} Oct 30 13:42:18.199253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.199127 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:42:18.199253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.199150 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" event={"ID":"56ffe634-75ee-4ed1-abea-7dda3aad4a0a","Type":"ContainerStarted","Data":"8f7e9dfb059367506666379d24bf2df49edc581ff93ef92929a1d867ec61473c"} Oct 30 13:42:18.201304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.201272 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" event={"ID":"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd","Type":"ContainerStarted","Data":"ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034"} Oct 30 13:42:18.201433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.201332 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" event={"ID":"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd","Type":"ContainerStarted","Data":"18d2ad9e13e2527e891af2a390286a6f021efe815d4da3ff9a07451e8327ba7f"} Oct 30 13:42:18.201433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.201358 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:42:18.242836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.242727 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" podStartSLOduration=2.242708298 podStartE2EDuration="2.242708298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:18.236939898 +0000 UTC m=+3045.052134322" watchObservedRunningTime="2026-10-30 13:42:18.242708298 +0000 UTC m=+3045.057902723" Oct 30 13:42:18.278836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.278782 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" podStartSLOduration=2.278768084 podStartE2EDuration="2.278768084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:18.273814512 +0000 UTC m=+3045.089008931" watchObservedRunningTime="2026-10-30 13:42:18.278768084 +0000 UTC m=+3045.093962507" Oct 30 13:42:18.352336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.352288 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:18.701646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:18.701598 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk"] Oct 30 13:42:19.213276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:19.213233 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" event={"ID":"0065d53d-a758-49dc-a2a5-9b50710eac2b","Type":"ContainerStarted","Data":"77a96e261f88b8aa8bfe2da87d8fcfd6623ecb2c22f662665d025a29bf1d0aa5"} Oct 30 13:42:19.213546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:19.213358 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" event={"ID":"0065d53d-a758-49dc-a2a5-9b50710eac2b","Type":"ContainerStarted","Data":"58fb12768721c9e98c9a3ce5d7f0c25ba1acafeaef9f95b5c6856f8986c2d7f5"} Oct 30 13:42:19.257931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:19.257875 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" podStartSLOduration=2.257862139 podStartE2EDuration="2.257862139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:19.251530765 +0000 UTC m=+3046.066725214" watchObservedRunningTime="2026-10-30 13:42:19.257862139 +0000 UTC m=+3046.073056563" Oct 30 13:42:19.917234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:19.917195 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:20.223545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:20.223514 5049 generic.go:356] "Generic (PLEG): container finished" podID="0065d53d-a758-49dc-a2a5-9b50710eac2b" containerID="77a96e261f88b8aa8bfe2da87d8fcfd6623ecb2c22f662665d025a29bf1d0aa5" exitCode=0 Oct 30 13:42:20.223764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:20.223604 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" event={"ID":"0065d53d-a758-49dc-a2a5-9b50710eac2b","Type":"ContainerDied","Data":"77a96e261f88b8aa8bfe2da87d8fcfd6623ecb2c22f662665d025a29bf1d0aa5"} Oct 30 13:42:21.431095 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.431060 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:21.440623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.440594 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:21.509721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.509625 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-xks-ca\") pod \"0065d53d-a758-49dc-a2a5-9b50710eac2b\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " Oct 30 13:42:21.509721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.509699 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0065d53d-a758-49dc-a2a5-9b50710eac2b-kube-api-access-qx2wv\" (UniqueName: \"kubernetes.io/projected/0065d53d-a758-49dc-a2a5-9b50710eac2b-kube-api-access-qx2wv\") pod \"0065d53d-a758-49dc-a2a5-9b50710eac2b\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " Oct 30 13:42:21.509957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.509862 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-openshift-ca\") pod \"0065d53d-a758-49dc-a2a5-9b50710eac2b\" (UID: \"0065d53d-a758-49dc-a2a5-9b50710eac2b\") " Oct 30 13:42:21.511144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.511112 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-openshift-ca" pod "0065d53d-a758-49dc-a2a5-9b50710eac2b" (UID: "0065d53d-a758-49dc-a2a5-9b50710eac2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:21.512810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.512777 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-xks-ca" pod "0065d53d-a758-49dc-a2a5-9b50710eac2b" (UID: "0065d53d-a758-49dc-a2a5-9b50710eac2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:21.516811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.516785 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0065d53d-a758-49dc-a2a5-9b50710eac2b-kube-api-access-qx2wv" pod "0065d53d-a758-49dc-a2a5-9b50710eac2b" (UID: "0065d53d-a758-49dc-a2a5-9b50710eac2b"). InnerVolumeSpecName "kube-api-access-qx2wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:21.612499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.612461 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:21.612499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.612498 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx2wv\" (UniqueName: \"kubernetes.io/projected/0065d53d-a758-49dc-a2a5-9b50710eac2b-kube-api-access-qx2wv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:21.612725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.612512 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0065d53d-a758-49dc-a2a5-9b50710eac2b-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:21.644195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.644158 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:21.796773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:21.796747 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:22.060036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:22.059950 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:22.194433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:22.194391 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6"] Oct 30 13:42:22.230390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:22.221377 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c54f0f-4l8q6"] Oct 30 13:42:22.230390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:22.228812 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:22.245383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:22.245345 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" event={"ID":"0065d53d-a758-49dc-a2a5-9b50710eac2b","Type":"ContainerDied","Data":"58fb12768721c9e98c9a3ce5d7f0c25ba1acafeaef9f95b5c6856f8986c2d7f5"} Oct 30 13:42:22.245383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:22.245384 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk" Oct 30 13:42:22.245638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:22.245413 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58fb12768721c9e98c9a3ce5d7f0c25ba1acafeaef9f95b5c6856f8986c2d7f5" Oct 30 13:42:22.388214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:22.388120 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:22.715376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:22.715272 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:23.339890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:23.339854 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852" path="/var/lib/kubelet/pods/3ea9d9ea-4bd0-48da-b94a-a9a0db4e9852/volumes" Oct 30 13:42:24.883345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:24.872495 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:26.679595 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:42:26.679557 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:42:26.786682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:26.786596 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:28.375607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:28.375571 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:28.970479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:28.970447 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:29.114912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:29.114873 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt"] Oct 30 13:42:29.124531 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:29.124495 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cb58b8-8pnwt"] Oct 30 13:42:29.145138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:29.145102 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:29.215928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:29.215897 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:42:29.216136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:29.216064 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:42:29.320363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:29.320316 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:29.343287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:29.343254 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5287534f-89cd-4048-87a5-4d3039a9ac7d" path="/var/lib/kubelet/pods/5287534f-89cd-4048-87a5-4d3039a9ac7d/volumes" Oct 30 13:42:29.806129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:29.806093 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:30.148140 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.148046 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:30.310283 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.310251 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:30.413672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.413577 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk"] Oct 30 13:42:30.413879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.413827 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" containerID="cri-o://b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb" gracePeriod=30 Oct 30 13:42:30.492224 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.492192 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:30.967087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.967047 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn"] Oct 30 13:42:30.967946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.967927 5049 cpu_manager.go:414] "containerMap: removing container" podUID="0065d53d-a758-49dc-a2a5-9b50710eac2b" containerName="revoke-keys" Oct 30 13:42:30.968018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.967947 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0065d53d-a758-49dc-a2a5-9b50710eac2b" containerName="revoke-keys" Oct 30 13:42:30.968105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.968096 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="0065d53d-a758-49dc-a2a5-9b50710eac2b" containerName="revoke-keys" Oct 30 13:42:30.973399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.973382 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:30.987610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.987586 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:42:30.993467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.993438 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn"] Oct 30 13:42:30.997783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:30.997754 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:42:31.104420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.104381 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:31.104624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.104439 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:31.104765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.104732 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v67z9\" (UniqueName: \"kubernetes.io/projected/47807a8c-a577-4e49-98b4-25b97684ec53-kube-api-access-v67z9\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:31.164810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.164767 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:31.206748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.206706 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v67z9\" (UniqueName: \"kubernetes.io/projected/47807a8c-a577-4e49-98b4-25b97684ec53-kube-api-access-v67z9\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:31.206948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.206903 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:31.207021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.206948 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:31.207776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.207748 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:31.209608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.209586 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:31.229175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.229104 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v67z9\" (UniqueName: \"kubernetes.io/projected/47807a8c-a577-4e49-98b4-25b97684ec53-kube-api-access-v67z9\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:31.288664 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.288387 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:31.367942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.367911 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:31.518063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.517972 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:31.725427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.712490 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn"] Oct 30 13:42:31.936771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:31.936736 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:32.181173 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.181134 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:32.186111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.186082 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk"] Oct 30 13:42:32.195731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.195702 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8d3b8-nvmfk"] Oct 30 13:42:32.254075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.254044 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz"] Oct 30 13:42:32.258935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.258911 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.283444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.280344 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz"] Oct 30 13:42:32.340535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.340381 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:32.348596 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.348567 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.348790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.348698 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xhkh\" (UniqueName: \"kubernetes.io/projected/fa5234bd-4c0c-4c54-8780-871ad4d9249d-kube-api-access-4xhkh\") pod \"maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.348790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.348782 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.371551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.371510 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" event={"ID":"47807a8c-a577-4e49-98b4-25b97684ec53","Type":"ContainerStarted","Data":"d40438c6b71243c4fbd7406aede3faa55b4136880c108b7a919d53845a6526a6"} Oct 30 13:42:32.371859 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.371835 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" event={"ID":"47807a8c-a577-4e49-98b4-25b97684ec53","Type":"ContainerStarted","Data":"da10bf006984900fa7d3af63f080ba69f4a78c82e600cde1ff7fcd86d2dab1e2"} Oct 30 13:42:32.412965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.412906 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" podStartSLOduration=2.412892942 podStartE2EDuration="2.412892942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:32.410017949 +0000 UTC m=+3059.225212372" watchObservedRunningTime="2026-10-30 13:42:32.412892942 +0000 UTC m=+3059.228087385" Oct 30 13:42:32.449971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.449867 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.449971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.449942 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.450446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.450105 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xhkh\" (UniqueName: \"kubernetes.io/projected/fa5234bd-4c0c-4c54-8780-871ad4d9249d-kube-api-access-4xhkh\") pod \"maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.450818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.450784 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.453038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.453008 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.472147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.472111 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xhkh\" (UniqueName: \"kubernetes.io/projected/fa5234bd-4c0c-4c54-8780-871ad4d9249d-kube-api-access-4xhkh\") pod \"maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.575484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.575445 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:32.743556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.743467 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:32.900031 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:32.900008 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz"] Oct 30 13:42:32.902790 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:42:32.902759 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5234bd_4c0c_4c54_8780_871ad4d9249d.slice/crio-3e7584cba483014990690a6e0f4bf5a7307a2653a8d7d0e161c45123d1408c05 WatchSource:0}: Error finding container 3e7584cba483014990690a6e0f4bf5a7307a2653a8d7d0e161c45123d1408c05: Status 404 returned error can't find the container with id 3e7584cba483014990690a6e0f4bf5a7307a2653a8d7d0e161c45123d1408c05 Oct 30 13:42:33.340078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:33.340039 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0065d53d-a758-49dc-a2a5-9b50710eac2b" path="/var/lib/kubelet/pods/0065d53d-a758-49dc-a2a5-9b50710eac2b/volumes" Oct 30 13:42:33.382146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:33.382051 5049 generic.go:356] "Generic (PLEG): container finished" podID="47807a8c-a577-4e49-98b4-25b97684ec53" containerID="d40438c6b71243c4fbd7406aede3faa55b4136880c108b7a919d53845a6526a6" exitCode=0 Oct 30 13:42:33.382311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:33.382136 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" event={"ID":"47807a8c-a577-4e49-98b4-25b97684ec53","Type":"ContainerDied","Data":"d40438c6b71243c4fbd7406aede3faa55b4136880c108b7a919d53845a6526a6"} Oct 30 13:42:33.384078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:33.384058 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" event={"ID":"fa5234bd-4c0c-4c54-8780-871ad4d9249d","Type":"ContainerStarted","Data":"150b49ff26b9f5e214d11c82dab2a6fbf560d6c29a86b0357683b81192d85514"} Oct 30 13:42:33.384167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:33.384100 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" event={"ID":"fa5234bd-4c0c-4c54-8780-871ad4d9249d","Type":"ContainerStarted","Data":"3e7584cba483014990690a6e0f4bf5a7307a2653a8d7d0e161c45123d1408c05"} Oct 30 13:42:33.465368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:33.461612 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" podStartSLOduration=1.461593423 podStartE2EDuration="1.461593423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:33.459290633 +0000 UTC m=+3060.274485057" watchObservedRunningTime="2026-10-30 13:42:33.461593423 +0000 UTC m=+3060.276787846" Oct 30 13:42:34.311611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.311573 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:34.394877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.394841 5049 generic.go:356] "Generic (PLEG): container finished" podID="fa5234bd-4c0c-4c54-8780-871ad4d9249d" containerID="150b49ff26b9f5e214d11c82dab2a6fbf560d6c29a86b0357683b81192d85514" exitCode=0 Oct 30 13:42:34.395361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.394912 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" event={"ID":"fa5234bd-4c0c-4c54-8780-871ad4d9249d","Type":"ContainerDied","Data":"150b49ff26b9f5e214d11c82dab2a6fbf560d6c29a86b0357683b81192d85514"} Oct 30 13:42:34.573942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.573879 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:34.685796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.685764 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47807a8c-a577-4e49-98b4-25b97684ec53-kube-api-access-v67z9\" (UniqueName: \"kubernetes.io/projected/47807a8c-a577-4e49-98b4-25b97684ec53-kube-api-access-v67z9\") pod \"47807a8c-a577-4e49-98b4-25b97684ec53\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " Oct 30 13:42:34.685999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.685811 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-xks-ca\") pod \"47807a8c-a577-4e49-98b4-25b97684ec53\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " Oct 30 13:42:34.685999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.685905 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-openshift-ca\") pod \"47807a8c-a577-4e49-98b4-25b97684ec53\" (UID: \"47807a8c-a577-4e49-98b4-25b97684ec53\") " Oct 30 13:42:34.686303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.686268 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-openshift-ca" pod "47807a8c-a577-4e49-98b4-25b97684ec53" (UID: "47807a8c-a577-4e49-98b4-25b97684ec53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:34.688279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.688250 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-xks-ca" pod "47807a8c-a577-4e49-98b4-25b97684ec53" (UID: "47807a8c-a577-4e49-98b4-25b97684ec53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:34.688423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.688355 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47807a8c-a577-4e49-98b4-25b97684ec53-kube-api-access-v67z9" pod "47807a8c-a577-4e49-98b4-25b97684ec53" (UID: "47807a8c-a577-4e49-98b4-25b97684ec53"). InnerVolumeSpecName "kube-api-access-v67z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:34.787148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.787116 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v67z9\" (UniqueName: \"kubernetes.io/projected/47807a8c-a577-4e49-98b4-25b97684ec53-kube-api-access-v67z9\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:34.787148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.787144 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:34.787148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:34.787157 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47807a8c-a577-4e49-98b4-25b97684ec53-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.404935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.404907 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" Oct 30 13:42:35.405360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.404951 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn" event={"ID":"47807a8c-a577-4e49-98b4-25b97684ec53","Type":"ContainerDied","Data":"da10bf006984900fa7d3af63f080ba69f4a78c82e600cde1ff7fcd86d2dab1e2"} Oct 30 13:42:35.405360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.404997 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da10bf006984900fa7d3af63f080ba69f4a78c82e600cde1ff7fcd86d2dab1e2" Oct 30 13:42:35.585421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.585394 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:35.595862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.595835 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-openshift-ca\") pod \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " Oct 30 13:42:35.595991 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.595977 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5234bd-4c0c-4c54-8780-871ad4d9249d-kube-api-access-4xhkh\" (UniqueName: \"kubernetes.io/projected/fa5234bd-4c0c-4c54-8780-871ad4d9249d-kube-api-access-4xhkh\") pod \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " Oct 30 13:42:35.596036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.596007 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-xks-ca\") pod \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\" (UID: \"fa5234bd-4c0c-4c54-8780-871ad4d9249d\") " Oct 30 13:42:35.596160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.596136 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-openshift-ca" pod "fa5234bd-4c0c-4c54-8780-871ad4d9249d" (UID: "fa5234bd-4c0c-4c54-8780-871ad4d9249d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:35.596622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.596599 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.598084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.598055 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5234bd-4c0c-4c54-8780-871ad4d9249d-kube-api-access-4xhkh" pod "fa5234bd-4c0c-4c54-8780-871ad4d9249d" (UID: "fa5234bd-4c0c-4c54-8780-871ad4d9249d"). InnerVolumeSpecName "kube-api-access-4xhkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:35.598084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.598063 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-xks-ca" pod "fa5234bd-4c0c-4c54-8780-871ad4d9249d" (UID: "fa5234bd-4c0c-4c54-8780-871ad4d9249d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:35.698940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.698907 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xhkh\" (UniqueName: \"kubernetes.io/projected/fa5234bd-4c0c-4c54-8780-871ad4d9249d-kube-api-access-4xhkh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.698940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.698942 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa5234bd-4c0c-4c54-8780-871ad4d9249d-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.728549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.728519 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:42:35.805250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.805214 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-certs\") pod \"02eb2218-687c-46f4-8627-a831c5f074fe\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " Oct 30 13:42:35.805519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.805500 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-credential-socket\") pod \"02eb2218-687c-46f4-8627-a831c5f074fe\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " Oct 30 13:42:35.805669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.805650 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-kube-api-access-tctq6\" (UniqueName: \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-kube-api-access-tctq6\") pod \"02eb2218-687c-46f4-8627-a831c5f074fe\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " Oct 30 13:42:35.805855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.805835 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02eb2218-687c-46f4-8627-a831c5f074fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02eb2218-687c-46f4-8627-a831c5f074fe-istio-podinfo\") pod \"02eb2218-687c-46f4-8627-a831c5f074fe\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " Oct 30 13:42:35.806027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.806010 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02eb2218-687c-46f4-8627-a831c5f074fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02eb2218-687c-46f4-8627-a831c5f074fe-istiod-ca-cert\") pod \"02eb2218-687c-46f4-8627-a831c5f074fe\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " Oct 30 13:42:35.806150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.806134 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-socket\") pod \"02eb2218-687c-46f4-8627-a831c5f074fe\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " Oct 30 13:42:35.806465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.806315 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-envoy\") pod \"02eb2218-687c-46f4-8627-a831c5f074fe\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " Oct 30 13:42:35.806671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.806651 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-data\") pod \"02eb2218-687c-46f4-8627-a831c5f074fe\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " Oct 30 13:42:35.806879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.806863 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-istio-token\" (UniqueName: \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-istio-token\") pod \"02eb2218-687c-46f4-8627-a831c5f074fe\" (UID: \"02eb2218-687c-46f4-8627-a831c5f074fe\") " Oct 30 13:42:35.807964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.807942 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-socket" pod "02eb2218-687c-46f4-8627-a831c5f074fe" (UID: "02eb2218-687c-46f4-8627-a831c5f074fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:35.808389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.808370 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02eb2218-687c-46f4-8627-a831c5f074fe-istiod-ca-cert" pod "02eb2218-687c-46f4-8627-a831c5f074fe" (UID: "02eb2218-687c-46f4-8627-a831c5f074fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:35.808725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.808707 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-credential-socket" pod "02eb2218-687c-46f4-8627-a831c5f074fe" (UID: "02eb2218-687c-46f4-8627-a831c5f074fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:35.809001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.808982 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-certs" pod "02eb2218-687c-46f4-8627-a831c5f074fe" (UID: "02eb2218-687c-46f4-8627-a831c5f074fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:35.809613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.809592 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-data" pod "02eb2218-687c-46f4-8627-a831c5f074fe" (UID: "02eb2218-687c-46f4-8627-a831c5f074fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:35.810654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.810637 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.810791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.810776 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.810890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.810876 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.810990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.810976 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02eb2218-687c-46f4-8627-a831c5f074fe-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.811092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.811080 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.813940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.813914 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-istio-token" pod "02eb2218-687c-46f4-8627-a831c5f074fe" (UID: "02eb2218-687c-46f4-8627-a831c5f074fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:35.814046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.814013 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-envoy" pod "02eb2218-687c-46f4-8627-a831c5f074fe" (UID: "02eb2218-687c-46f4-8627-a831c5f074fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:35.815180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.814889 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-kube-api-access-tctq6" pod "02eb2218-687c-46f4-8627-a831c5f074fe" (UID: "02eb2218-687c-46f4-8627-a831c5f074fe"). InnerVolumeSpecName "kube-api-access-tctq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:35.815653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.815619 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02eb2218-687c-46f4-8627-a831c5f074fe-istio-podinfo" pod "02eb2218-687c-46f4-8627-a831c5f074fe" (UID: "02eb2218-687c-46f4-8627-a831c5f074fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:42:35.888625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.888536 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:35.912852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.912808 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.912852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.912850 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tctq6\" (UniqueName: \"kubernetes.io/projected/02eb2218-687c-46f4-8627-a831c5f074fe-kube-api-access-tctq6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.912852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.912863 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02eb2218-687c-46f4-8627-a831c5f074fe-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:35.913179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:35.912874 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02eb2218-687c-46f4-8627-a831c5f074fe-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:36.078562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.078530 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:36.243805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.243701 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:36.396342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.391229 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:36.435363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.435284 5049 generic.go:356] "Generic (PLEG): container finished" podID="02eb2218-687c-46f4-8627-a831c5f074fe" containerID="b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb" exitCode=0 Oct 30 13:42:36.435866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.435411 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" event={"ID":"02eb2218-687c-46f4-8627-a831c5f074fe","Type":"ContainerDied","Data":"b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb"} Oct 30 13:42:36.435866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.435466 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" event={"ID":"02eb2218-687c-46f4-8627-a831c5f074fe","Type":"ContainerDied","Data":"ea66e16ece92f6cabc9a238d6b9c69b4a1bd91a8a9b2ceea3a07ff025b8fe216"} Oct 30 13:42:36.435866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.435499 5049 scope.go:122] "RemoveContainer" containerID="b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb" Oct 30 13:42:36.436352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.436171 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk" Oct 30 13:42:36.443853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.443810 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" event={"ID":"fa5234bd-4c0c-4c54-8780-871ad4d9249d","Type":"ContainerDied","Data":"3e7584cba483014990690a6e0f4bf5a7307a2653a8d7d0e161c45123d1408c05"} Oct 30 13:42:36.447545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.447527 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e7584cba483014990690a6e0f4bf5a7307a2653a8d7d0e161c45123d1408c05" Oct 30 13:42:36.448085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.444033 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz" Oct 30 13:42:36.476397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.475445 5049 scope.go:122] "RemoveContainer" containerID="6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177" Oct 30 13:42:36.493611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.493584 5049 scope.go:122] "RemoveContainer" containerID="b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb" Oct 30 13:42:36.494064 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:42:36.494008 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb\": container with ID starting with b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb not found: ID does not exist" containerID="b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb" Oct 30 13:42:36.494141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.494056 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb"} err="failed to get container status \"b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb\": rpc error: code = NotFound desc = could not find container \"b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb\": container with ID starting with b8fc399d1b1499752103068fcd202ef3388c5997cec997c0e9c3b1810dc567fb not found: ID does not exist" Oct 30 13:42:36.494141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.494082 5049 scope.go:122] "RemoveContainer" containerID="6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177" Oct 30 13:42:36.495518 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:42:36.495486 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177\": container with ID starting with 6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177 not found: ID does not exist" containerID="6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177" Oct 30 13:42:36.495612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.495531 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177"} err="failed to get container status \"6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177\": rpc error: code = NotFound desc = could not find container \"6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177\": container with ID starting with 6a6332452a7f05c1b4c8d1ba5e92d6e7e441099e353cde5feadddd9859b76177 not found: ID does not exist" Oct 30 13:42:36.548469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.548427 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk"] Oct 30 13:42:36.559151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.559122 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c54f0f-openshift-default-84769d4c97-wlqkk"] Oct 30 13:42:36.575683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.575632 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:36.661003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.660965 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v"] Oct 30 13:42:36.661220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.661197 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerName="istio-proxy" containerID="cri-o://337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887" gracePeriod=30 Oct 30 13:42:36.767028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.766948 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:36.887436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.880518 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v"] Oct 30 13:42:36.903578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.903531 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf3c79-l9x4v"] Oct 30 13:42:36.952360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:36.952051 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:37.339595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:37.339565 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" path="/var/lib/kubelet/pods/02eb2218-687c-46f4-8627-a831c5f074fe/volumes" Oct 30 13:42:37.340083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:37.340066 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f165eb17-95a5-4da8-9627-b858f43b1100" path="/var/lib/kubelet/pods/f165eb17-95a5-4da8-9627-b858f43b1100/volumes" Oct 30 13:42:37.358779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:37.358753 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:37.691389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:37.691288 5049 scope.go:122] "RemoveContainer" containerID="14cffb3ce43bac2f65bfaa9470b7b3acef7d41c449437c7da99e87a72379533a" Oct 30 13:42:37.702872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:37.702847 5049 scope.go:122] "RemoveContainer" containerID="e6d98474fb97222ede8acc84609a45f3879bee804e8ce4dd4410cdb756193c50" Oct 30 13:42:38.875746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:38.875705 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:39.988022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:39.987973 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:40.152524 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.152493 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:40.367987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.367942 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:40.868314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.868276 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8"] Oct 30 13:42:40.869352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869309 5049 cpu_manager.go:414] "containerMap: removing container" podUID="47807a8c-a577-4e49-98b4-25b97684ec53" containerName="revoke-keys" Oct 30 13:42:40.869352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869348 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47807a8c-a577-4e49-98b4-25b97684ec53" containerName="revoke-keys" Oct 30 13:42:40.869609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869395 5049 cpu_manager.go:414] "containerMap: removing container" podUID="fa5234bd-4c0c-4c54-8780-871ad4d9249d" containerName="revoke-keys" Oct 30 13:42:40.869609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869403 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5234bd-4c0c-4c54-8780-871ad4d9249d" containerName="revoke-keys" Oct 30 13:42:40.869609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869444 5049 cpu_manager.go:414] "containerMap: removing container" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" Oct 30 13:42:40.869609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869451 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" Oct 30 13:42:40.869609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869458 5049 cpu_manager.go:414] "containerMap: removing container" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" Oct 30 13:42:40.869609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869464 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" Oct 30 13:42:40.869609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869611 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="47807a8c-a577-4e49-98b4-25b97684ec53" containerName="revoke-keys" Oct 30 13:42:40.870010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869630 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5234bd-4c0c-4c54-8780-871ad4d9249d" containerName="revoke-keys" Oct 30 13:42:40.870010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869648 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" Oct 30 13:42:40.870010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.869659 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="02eb2218-687c-46f4-8627-a831c5f074fe" containerName="istio-proxy" Oct 30 13:42:40.875649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.875621 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:40.898718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.898691 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:42:40.898860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.898785 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:42:40.901883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.901854 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:40.905188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.905157 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:40.905393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.905366 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52fz\" (UniqueName: \"kubernetes.io/projected/73504fcb-46a4-451f-ade1-c9b5edfa0940-kube-api-access-j52fz\") pod \"maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:40.905496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.905411 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:40.909715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:40.909689 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8"] Oct 30 13:42:41.007048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.007012 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:41.007564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.007131 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52fz\" (UniqueName: \"kubernetes.io/projected/73504fcb-46a4-451f-ade1-c9b5edfa0940-kube-api-access-j52fz\") pod \"maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:41.007564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.007152 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:41.008030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.008004 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:41.009898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.009873 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:41.028782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.028755 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52fz\" (UniqueName: \"kubernetes.io/projected/73504fcb-46a4-451f-ade1-c9b5edfa0940-kube-api-access-j52fz\") pod \"maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:41.190400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.190265 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:41.353074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.353038 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r"] Oct 30 13:42:41.377117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.377084 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f"] Oct 30 13:42:41.504851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.504817 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8"] Oct 30 13:42:41.510087 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:42:41.510053 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73504fcb_46a4_451f_ade1_c9b5edfa0940.slice/crio-434e65b0811e211f77eb7f5bb78a0197f76f997c5343fbd420b3f34445bb7142 WatchSource:0}: Error finding container 434e65b0811e211f77eb7f5bb78a0197f76f997c5343fbd420b3f34445bb7142: Status 404 returned error can't find the container with id 434e65b0811e211f77eb7f5bb78a0197f76f997c5343fbd420b3f34445bb7142 Oct 30 13:42:41.534342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.530713 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:41.684518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:41.684487 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:42.097129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.097104 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:42:42.135190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135156 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-envoy\") pod \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " Oct 30 13:42:42.135418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135228 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-socket\") pod \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " Oct 30 13:42:42.135418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135254 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-kube-api-access-6hjlc\" (UniqueName: \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-kube-api-access-6hjlc\") pod \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " Oct 30 13:42:42.135418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135293 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-data\") pod \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " Oct 30 13:42:42.135418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135350 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-podinfo\") pod \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " Oct 30 13:42:42.135418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135379 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-token\" (UniqueName: \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-token\") pod \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " Oct 30 13:42:42.135748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135465 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istiod-ca-cert\") pod \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " Oct 30 13:42:42.135748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135544 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-socket" pod "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" (UID: "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:42.135748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135574 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-certs\") pod \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " Oct 30 13:42:42.135748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135673 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-credential-socket\") pod \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\" (UID: \"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273\") " Oct 30 13:42:42.135972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.135893 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istiod-ca-cert" pod "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" (UID: "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:42.136089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.136066 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-data" pod "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" (UID: "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:42.136154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.136113 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-credential-socket" pod "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" (UID: "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:42.136210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.136180 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-certs" pod "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" (UID: "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:42.136420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.136394 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.136549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.136427 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.136549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.136441 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.136549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.136458 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.136549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.136473 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.137809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.137784 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-token" pod "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" (UID: "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:42.137980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.137947 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-envoy" pod "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" (UID: "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:42.137980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.137960 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-kube-api-access-6hjlc" pod "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" (UID: "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273"). InnerVolumeSpecName "kube-api-access-6hjlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:42.138109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.138026 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-podinfo" pod "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" (UID: "eaa99dc8-e3a2-4395-a917-c6b1f6e5d273"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:42:42.237408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.237373 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.237408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.237404 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hjlc\" (UniqueName: \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-kube-api-access-6hjlc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.237696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.237420 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.237696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.237429 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.525824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.525775 5049 generic.go:356] "Generic (PLEG): container finished" podID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerID="337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887" exitCode=0 Oct 30 13:42:42.525995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.525846 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" Oct 30 13:42:42.525995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.525842 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" event={"ID":"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273","Type":"ContainerDied","Data":"337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887"} Oct 30 13:42:42.526212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.526003 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v" event={"ID":"eaa99dc8-e3a2-4395-a917-c6b1f6e5d273","Type":"ContainerDied","Data":"3f564d093fe4a573b8a59742bfae2b27cb94712c308f47eb27774061f6ad8a53"} Oct 30 13:42:42.526212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.526037 5049 scope.go:122] "RemoveContainer" containerID="337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887" Oct 30 13:42:42.528829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.528800 5049 generic.go:356] "Generic (PLEG): container finished" podID="73504fcb-46a4-451f-ade1-c9b5edfa0940" containerID="f43a5362c4bbe1842473c3a83bd3d3bdadbc3cd996124045b03afb4a9c4834bd" exitCode=0 Oct 30 13:42:42.528986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.528968 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" event={"ID":"73504fcb-46a4-451f-ade1-c9b5edfa0940","Type":"ContainerDied","Data":"f43a5362c4bbe1842473c3a83bd3d3bdadbc3cd996124045b03afb4a9c4834bd"} Oct 30 13:42:42.529060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.529018 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" event={"ID":"73504fcb-46a4-451f-ade1-c9b5edfa0940","Type":"ContainerStarted","Data":"434e65b0811e211f77eb7f5bb78a0197f76f997c5343fbd420b3f34445bb7142"} Oct 30 13:42:42.547159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.547111 5049 scope.go:122] "RemoveContainer" containerID="337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887" Oct 30 13:42:42.547896 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:42:42.547865 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887\": container with ID starting with 337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887 not found: ID does not exist" containerID="337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887" Oct 30 13:42:42.548010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.547906 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887"} err="failed to get container status \"337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887\": rpc error: code = NotFound desc = could not find container \"337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887\": container with ID starting with 337b538d0300621be4e0d227cf6b183d2a650f05c00c4581e243183c18875887 not found: ID does not exist" Oct 30 13:42:42.612048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.612014 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v"] Oct 30 13:42:42.621669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.621637 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-mjw8v"] Oct 30 13:42:42.991414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:42.987562 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:43.352225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.352188 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" path="/var/lib/kubelet/pods/eaa99dc8-e3a2-4395-a917-c6b1f6e5d273/volumes" Oct 30 13:42:43.719714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.719682 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:43.760598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.760565 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-openshift-ca\") pod \"73504fcb-46a4-451f-ade1-c9b5edfa0940\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " Oct 30 13:42:43.760795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.760634 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-xks-ca\") pod \"73504fcb-46a4-451f-ade1-c9b5edfa0940\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " Oct 30 13:42:43.760867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.760764 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73504fcb-46a4-451f-ade1-c9b5edfa0940-kube-api-access-j52fz\" (UniqueName: \"kubernetes.io/projected/73504fcb-46a4-451f-ade1-c9b5edfa0940-kube-api-access-j52fz\") pod \"73504fcb-46a4-451f-ade1-c9b5edfa0940\" (UID: \"73504fcb-46a4-451f-ade1-c9b5edfa0940\") " Oct 30 13:42:43.761018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.760990 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-openshift-ca" pod "73504fcb-46a4-451f-ade1-c9b5edfa0940" (UID: "73504fcb-46a4-451f-ade1-c9b5edfa0940"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:43.761470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.761443 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:43.763565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.763542 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73504fcb-46a4-451f-ade1-c9b5edfa0940-kube-api-access-j52fz" pod "73504fcb-46a4-451f-ade1-c9b5edfa0940" (UID: "73504fcb-46a4-451f-ade1-c9b5edfa0940"). InnerVolumeSpecName "kube-api-access-j52fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:43.764940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.764902 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-xks-ca" pod "73504fcb-46a4-451f-ade1-c9b5edfa0940" (UID: "73504fcb-46a4-451f-ade1-c9b5edfa0940"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:43.799796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.799677 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:43.863038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.863006 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52fz\" (UniqueName: \"kubernetes.io/projected/73504fcb-46a4-451f-ade1-c9b5edfa0940-kube-api-access-j52fz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:43.863038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.863036 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73504fcb-46a4-451f-ade1-c9b5edfa0940-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:43.966451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:43.966361 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:44.019850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.019808 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv"] Oct 30 13:42:44.020111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.020055 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" containerID="cri-o://154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d" gracePeriod=30 Oct 30 13:42:44.185350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.181925 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:44.550308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.550277 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" Oct 30 13:42:44.550724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.550314 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8" event={"ID":"73504fcb-46a4-451f-ade1-c9b5edfa0940","Type":"ContainerDied","Data":"434e65b0811e211f77eb7f5bb78a0197f76f997c5343fbd420b3f34445bb7142"} Oct 30 13:42:44.550724 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.550394 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="434e65b0811e211f77eb7f5bb78a0197f76f997c5343fbd420b3f34445bb7142" Oct 30 13:42:44.562367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.562336 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc"] Oct 30 13:42:44.563135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.563114 5049 cpu_manager.go:414] "containerMap: removing container" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerName="istio-proxy" Oct 30 13:42:44.563259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.563136 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerName="istio-proxy" Oct 30 13:42:44.563259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.563186 5049 cpu_manager.go:414] "containerMap: removing container" podUID="73504fcb-46a4-451f-ade1-c9b5edfa0940" containerName="revoke-keys" Oct 30 13:42:44.563259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.563195 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73504fcb-46a4-451f-ade1-c9b5edfa0940" containerName="revoke-keys" Oct 30 13:42:44.563466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.563443 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa99dc8-e3a2-4395-a917-c6b1f6e5d273" containerName="istio-proxy" Oct 30 13:42:44.563517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.563468 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="73504fcb-46a4-451f-ade1-c9b5edfa0940" containerName="revoke-keys" Oct 30 13:42:44.567444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.567425 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:44.579730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.579707 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:42:44.579867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.579854 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:42:44.588811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.588787 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc"] Oct 30 13:42:44.693104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.693064 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:44.693315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.693247 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:44.693504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.693484 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw8qp\" (UniqueName: \"kubernetes.io/projected/662b2ff1-7704-4fd9-a418-c165b91f285c-kube-api-access-qw8qp\") pod \"maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:44.794914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.794877 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:44.795115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.794994 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:44.795218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.795199 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8qp\" (UniqueName: \"kubernetes.io/projected/662b2ff1-7704-4fd9-a418-c165b91f285c-kube-api-access-qw8qp\") pod \"maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:44.795816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.795797 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:44.797490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.797465 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:44.817127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.817060 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw8qp\" (UniqueName: \"kubernetes.io/projected/662b2ff1-7704-4fd9-a418-c165b91f285c-kube-api-access-qw8qp\") pod \"maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:44.881497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:44.881459 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:45.089017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:45.088862 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:45.214920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:45.214892 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc"] Oct 30 13:42:45.218191 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:42:45.218149 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662b2ff1_7704_4fd9_a418_c165b91f285c.slice/crio-c26e749e44c6d2798c7a2924dc127d8b77698ea381bc9a58ecf5d591d9c340bb WatchSource:0}: Error finding container c26e749e44c6d2798c7a2924dc127d8b77698ea381bc9a58ecf5d591d9c340bb: Status 404 returned error can't find the container with id c26e749e44c6d2798c7a2924dc127d8b77698ea381bc9a58ecf5d591d9c340bb Oct 30 13:42:45.560566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:45.560531 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" event={"ID":"662b2ff1-7704-4fd9-a418-c165b91f285c","Type":"ContainerStarted","Data":"31fb9c331999d7605ea98bab435eaee6d7a4d4b7097234de1fa404f01f2ea1a5"} Oct 30 13:42:45.561051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:45.560588 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" event={"ID":"662b2ff1-7704-4fd9-a418-c165b91f285c","Type":"ContainerStarted","Data":"c26e749e44c6d2798c7a2924dc127d8b77698ea381bc9a58ecf5d591d9c340bb"} Oct 30 13:42:45.597926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:45.597871 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" podStartSLOduration=1.597857187 podStartE2EDuration="1.597857187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:45.597084773 +0000 UTC m=+3072.412279197" watchObservedRunningTime="2026-10-30 13:42:45.597857187 +0000 UTC m=+3072.413051610" Oct 30 13:42:46.353494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:46.353430 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" podUID="feff1c90-5a44-4a9e-bef2-2992eeac9ee6" containerName="payload-processing" containerID="cri-o://727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2" gracePeriod=25 Oct 30 13:42:46.377361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:46.377310 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" podUID="819b0f8d-71fd-4fed-ae5d-d619a030ffc5" containerName="payload-pre-processing" containerID="cri-o://2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9" gracePeriod=25 Oct 30 13:42:46.570264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:46.570234 5049 generic.go:356] "Generic (PLEG): container finished" podID="662b2ff1-7704-4fd9-a418-c165b91f285c" containerID="31fb9c331999d7605ea98bab435eaee6d7a4d4b7097234de1fa404f01f2ea1a5" exitCode=0 Oct 30 13:42:46.570692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:46.570334 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" event={"ID":"662b2ff1-7704-4fd9-a418-c165b91f285c","Type":"ContainerDied","Data":"31fb9c331999d7605ea98bab435eaee6d7a4d4b7097234de1fa404f01f2ea1a5"} Oct 30 13:42:47.455667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.455624 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:47.663714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.657679 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:47.766928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.766900 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:47.846341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.846036 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:47.893341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.876938 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662b2ff1-7704-4fd9-a418-c165b91f285c-kube-api-access-qw8qp\" (UniqueName: \"kubernetes.io/projected/662b2ff1-7704-4fd9-a418-c165b91f285c-kube-api-access-qw8qp\") pod \"662b2ff1-7704-4fd9-a418-c165b91f285c\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " Oct 30 13:42:47.893341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.877007 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-openshift-ca\") pod \"662b2ff1-7704-4fd9-a418-c165b91f285c\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " Oct 30 13:42:47.893341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.877116 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-xks-ca\") pod \"662b2ff1-7704-4fd9-a418-c165b91f285c\" (UID: \"662b2ff1-7704-4fd9-a418-c165b91f285c\") " Oct 30 13:42:47.922352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.915901 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662b2ff1-7704-4fd9-a418-c165b91f285c-kube-api-access-qw8qp" pod "662b2ff1-7704-4fd9-a418-c165b91f285c" (UID: "662b2ff1-7704-4fd9-a418-c165b91f285c"). InnerVolumeSpecName "kube-api-access-qw8qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:47.942348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.935227 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-openshift-ca" pod "662b2ff1-7704-4fd9-a418-c165b91f285c" (UID: "662b2ff1-7704-4fd9-a418-c165b91f285c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:47.942348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.936022 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw8qp\" (UniqueName: \"kubernetes.io/projected/662b2ff1-7704-4fd9-a418-c165b91f285c-kube-api-access-qw8qp\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:47.942348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.936048 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:47.980115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:47.980061 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-xks-ca" pod "662b2ff1-7704-4fd9-a418-c165b91f285c" (UID: "662b2ff1-7704-4fd9-a418-c165b91f285c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:48.037532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:48.037447 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662b2ff1-7704-4fd9-a418-c165b91f285c-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:48.234959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:48.234927 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:48.457810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:48.457778 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz"] Oct 30 13:42:48.471259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:48.471215 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:48.478506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:48.478467 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78dc89-95gtz"] Oct 30 13:42:48.593066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:48.593032 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" Oct 30 13:42:48.593066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:48.593052 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc" event={"ID":"662b2ff1-7704-4fd9-a418-c165b91f285c","Type":"ContainerDied","Data":"c26e749e44c6d2798c7a2924dc127d8b77698ea381bc9a58ecf5d591d9c340bb"} Oct 30 13:42:48.593368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:48.593106 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26e749e44c6d2798c7a2924dc127d8b77698ea381bc9a58ecf5d591d9c340bb" Oct 30 13:42:48.629574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:48.629541 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:49.012985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.012941 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:49.315912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.315879 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:49.320142 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.320116 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:42:49.339487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.339452 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa5234bd-4c0c-4c54-8780-871ad4d9249d" path="/var/lib/kubelet/pods/fa5234bd-4c0c-4c54-8780-871ad4d9249d/volumes" Oct 30 13:42:49.357666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.357641 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-kube-api-access-r22b9\" (UniqueName: \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-kube-api-access-r22b9\") pod \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " Oct 30 13:42:49.357837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.357681 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-certs\") pod \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " Oct 30 13:42:49.357837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.357716 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-envoy\") pod \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " Oct 30 13:42:49.357837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.357751 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-token\" (UniqueName: \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-token\") pod \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " Oct 30 13:42:49.357837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.357775 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-credential-socket\") pod \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " Oct 30 13:42:49.357837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.357808 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-data\") pod \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " Oct 30 13:42:49.358233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.357844 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-podinfo\") pod \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " Oct 30 13:42:49.358233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358044 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istiod-ca-cert\") pod \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " Oct 30 13:42:49.358233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358077 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-socket\") pod \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\" (UID: \"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36\") " Oct 30 13:42:49.358233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358079 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-credential-socket" pod "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" (UID: "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:49.358233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358092 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-certs" pod "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" (UID: "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:49.358233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358204 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-data" pod "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" (UID: "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:49.358592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358545 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-socket" pod "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" (UID: "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:49.358592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358552 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istiod-ca-cert" pod "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" (UID: "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:49.358718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358701 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.358781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358723 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.358781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358736 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.358781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358749 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.358781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.358763 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.360135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.360109 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-kube-api-access-r22b9" pod "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" (UID: "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36"). InnerVolumeSpecName "kube-api-access-r22b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:49.360233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.360136 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-token" pod "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" (UID: "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:49.360233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.360172 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-envoy" pod "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" (UID: "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:49.360233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.360177 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-podinfo" pod "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" (UID: "5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:42:49.460358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.460305 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r22b9\" (UniqueName: \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-kube-api-access-r22b9\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.460358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.460356 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.460614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.460371 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.460614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.460385 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.603552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.603516 5049 generic.go:356] "Generic (PLEG): container finished" podID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerID="154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d" exitCode=0 Oct 30 13:42:49.603764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.603591 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" Oct 30 13:42:49.603764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.603605 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" event={"ID":"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36","Type":"ContainerDied","Data":"154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d"} Oct 30 13:42:49.603764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.603667 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv" event={"ID":"5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36","Type":"ContainerDied","Data":"1b08afe33f35722b8f80c7c92195b7935433da89b1340f408483478a256b7519"} Oct 30 13:42:49.603764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.603692 5049 scope.go:122] "RemoveContainer" containerID="154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d" Oct 30 13:42:49.619413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.619395 5049 scope.go:122] "RemoveContainer" containerID="f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e" Oct 30 13:42:49.632287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.632270 5049 scope.go:122] "RemoveContainer" containerID="154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d" Oct 30 13:42:49.632631 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:42:49.632604 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d\": container with ID starting with 154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d not found: ID does not exist" containerID="154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d" Oct 30 13:42:49.632732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.632638 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d"} err="failed to get container status \"154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d\": rpc error: code = NotFound desc = could not find container \"154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d\": container with ID starting with 154e9abf429d18427084a8ad2dd53eaf38fb263d1c3ce5164233dec55fd9110d not found: ID does not exist" Oct 30 13:42:49.632732 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.632655 5049 scope.go:122] "RemoveContainer" containerID="f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e" Oct 30 13:42:49.632920 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:42:49.632904 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e\": container with ID starting with f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e not found: ID does not exist" containerID="f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e" Oct 30 13:42:49.632968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.632925 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e"} err="failed to get container status \"f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e\": rpc error: code = NotFound desc = could not find container \"f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e\": container with ID starting with f97e8d910c48b1f923008049a527397e0f68833e6261afc4727308cf363a922e not found: ID does not exist" Oct 30 13:42:49.654801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.654775 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv"] Oct 30 13:42:49.659411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:49.659387 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-cf3c79-openshift-default-69c5b4fb45-jj9vv"] Oct 30 13:42:50.589671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:50.589609 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:51.340081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:51.340038 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" path="/var/lib/kubelet/pods/5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36/volumes" Oct 30 13:42:52.491661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:52.491622 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:53.978576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:53.978536 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w"] Oct 30 13:42:53.979068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:53.978789 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" podUID="fbcea945-63c3-4aae-8613-65944877613a" containerName="maas-api" containerID="cri-o://b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78" gracePeriod=30 Oct 30 13:42:54.088696 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.088659 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:54.286659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.286614 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:54.304217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.304192 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:42:54.432372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.432314 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-metrics-certs\") pod \"fbcea945-63c3-4aae-8613-65944877613a\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " Oct 30 13:42:54.432548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.432418 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbcea945-63c3-4aae-8613-65944877613a-kube-api-access-djtww\" (UniqueName: \"kubernetes.io/projected/fbcea945-63c3-4aae-8613-65944877613a-kube-api-access-djtww\") pod \"fbcea945-63c3-4aae-8613-65944877613a\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " Oct 30 13:42:54.432548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.432500 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbcea945-63c3-4aae-8613-65944877613a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcea945-63c3-4aae-8613-65944877613a-openshift-service-ca\") pod \"fbcea945-63c3-4aae-8613-65944877613a\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " Oct 30 13:42:54.432769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.432749 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-maas-api-tls\") pod \"fbcea945-63c3-4aae-8613-65944877613a\" (UID: \"fbcea945-63c3-4aae-8613-65944877613a\") " Oct 30 13:42:54.432963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.432932 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcea945-63c3-4aae-8613-65944877613a-openshift-service-ca" pod "fbcea945-63c3-4aae-8613-65944877613a" (UID: "fbcea945-63c3-4aae-8613-65944877613a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:54.433481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.433455 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcea945-63c3-4aae-8613-65944877613a-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.434923 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.434888 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcea945-63c3-4aae-8613-65944877613a-kube-api-access-djtww" pod "fbcea945-63c3-4aae-8613-65944877613a" (UID: "fbcea945-63c3-4aae-8613-65944877613a"). InnerVolumeSpecName "kube-api-access-djtww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:54.435117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.435086 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-metrics-certs" pod "fbcea945-63c3-4aae-8613-65944877613a" (UID: "fbcea945-63c3-4aae-8613-65944877613a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:54.435205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.435118 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-maas-api-tls" pod "fbcea945-63c3-4aae-8613-65944877613a" (UID: "fbcea945-63c3-4aae-8613-65944877613a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:54.445156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.445131 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:54.538833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.538734 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.538833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.538777 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djtww\" (UniqueName: \"kubernetes.io/projected/fbcea945-63c3-4aae-8613-65944877613a-kube-api-access-djtww\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.538833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.538792 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbcea945-63c3-4aae-8613-65944877613a-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.660202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.660167 5049 generic.go:356] "Generic (PLEG): container finished" podID="fbcea945-63c3-4aae-8613-65944877613a" containerID="b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78" exitCode=0 Oct 30 13:42:54.660415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.660222 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" Oct 30 13:42:54.660415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.660247 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" event={"ID":"fbcea945-63c3-4aae-8613-65944877613a","Type":"ContainerDied","Data":"b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78"} Oct 30 13:42:54.660415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.660306 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w" event={"ID":"fbcea945-63c3-4aae-8613-65944877613a","Type":"ContainerDied","Data":"d8822fa03942c03577f9d7ae11030ca3f257fe854f9c37277d60ffeb47a2c610"} Oct 30 13:42:54.660415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.660352 5049 scope.go:122] "RemoveContainer" containerID="b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78" Oct 30 13:42:54.672594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.672564 5049 scope.go:122] "RemoveContainer" containerID="b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78" Oct 30 13:42:54.672858 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:42:54.672838 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78\": container with ID starting with b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78 not found: ID does not exist" containerID="b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78" Oct 30 13:42:54.672910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.672866 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78"} err="failed to get container status \"b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78\": rpc error: code = NotFound desc = could not find container \"b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78\": container with ID starting with b3686b875a6032570c1a90ab589fd6782357a721f887cc935dc314def2e04e78 not found: ID does not exist" Oct 30 13:42:54.699939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.699916 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:54.708961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.708935 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w"] Oct 30 13:42:54.718522 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.718488 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4fb25-d94b54998-sl99w"] Oct 30 13:42:54.824414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.824380 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj"] Oct 30 13:42:54.829756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.829730 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b4fb25-7mxkj"] Oct 30 13:42:54.853687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:54.853647 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:55.015166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:55.015127 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:55.340306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:55.340264 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="438941e6-a769-4861-b36d-2d9561a156a2" path="/var/lib/kubelet/pods/438941e6-a769-4861-b36d-2d9561a156a2/volumes" Oct 30 13:42:55.340839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:55.340823 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbcea945-63c3-4aae-8613-65944877613a" path="/var/lib/kubelet/pods/fbcea945-63c3-4aae-8613-65944877613a/volumes" Oct 30 13:42:55.373089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:55.373058 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:55.639823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:55.639731 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" podUID="819b0f8d-71fd-4fed-ae5d-d619a030ffc5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:42:55.639974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:55.639879 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" podUID="feff1c90-5a44-4a9e-bef2-2992eeac9ee6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:42:56.211437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:56.211403 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:56.420573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:56.420544 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:56.636472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:56.634595 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:56.720933 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:42:56.720893 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:42:57.185477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.185443 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc"] Oct 30 13:42:57.186168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186154 5049 cpu_manager.go:414] "containerMap: removing container" podUID="662b2ff1-7704-4fd9-a418-c165b91f285c" containerName="revoke-keys" Oct 30 13:42:57.186225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186169 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662b2ff1-7704-4fd9-a418-c165b91f285c" containerName="revoke-keys" Oct 30 13:42:57.186225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186181 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" Oct 30 13:42:57.186225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186187 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" Oct 30 13:42:57.186225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186194 5049 cpu_manager.go:414] "containerMap: removing container" podUID="fbcea945-63c3-4aae-8613-65944877613a" containerName="maas-api" Oct 30 13:42:57.186225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186199 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcea945-63c3-4aae-8613-65944877613a" containerName="maas-api" Oct 30 13:42:57.186225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186210 5049 cpu_manager.go:414] "containerMap: removing container" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" Oct 30 13:42:57.186225 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186215 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" Oct 30 13:42:57.186473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186359 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="662b2ff1-7704-4fd9-a418-c165b91f285c" containerName="revoke-keys" Oct 30 13:42:57.186473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186372 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" Oct 30 13:42:57.186473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186380 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbcea945-63c3-4aae-8613-65944877613a" containerName="maas-api" Oct 30 13:42:57.186473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.186389 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcdf8ca-4321-4ccf-ab10-a0e53be8fd36" containerName="istio-proxy" Oct 30 13:42:57.191496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.191477 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.223455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.221391 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:42:57.223455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.221422 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:42:57.228606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.228575 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc"] Oct 30 13:42:57.284746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.284712 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.284933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.284777 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.284933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.284893 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmgp\" (UniqueName: \"kubernetes.io/projected/a5bf63e0-9560-4894-84b6-1d5c399783a8-kube-api-access-7zmgp\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.386570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.386540 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.386753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.386618 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.386753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.386675 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmgp\" (UniqueName: \"kubernetes.io/projected/a5bf63e0-9560-4894-84b6-1d5c399783a8-kube-api-access-7zmgp\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.387310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.387284 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.389577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.389549 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.413332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.413296 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmgp\" (UniqueName: \"kubernetes.io/projected/a5bf63e0-9560-4894-84b6-1d5c399783a8-kube-api-access-7zmgp\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.507771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.507690 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:42:57.870496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:57.866500 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc"] Oct 30 13:42:58.176370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:58.175851 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:58.712754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:58.712722 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" event={"ID":"a5bf63e0-9560-4894-84b6-1d5c399783a8","Type":"ContainerStarted","Data":"ec6b62b640e756be3e0f579ad22c628a2bf312749baa8c2f4c72e125426fa84f"} Oct 30 13:42:58.713193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:58.712780 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" event={"ID":"a5bf63e0-9560-4894-84b6-1d5c399783a8","Type":"ContainerStarted","Data":"2546c66fc69b5a74686101a0277c6104ef485d108b98d6af11cd4e9b81f74e23"} Oct 30 13:42:58.751743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:58.751690 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" podStartSLOduration=1.751677427 podStartE2EDuration="1.751677427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:58.749637728 +0000 UTC m=+3085.564832150" watchObservedRunningTime="2026-10-30 13:42:58.751677427 +0000 UTC m=+3085.566871851" Oct 30 13:42:59.723691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:59.723654 5049 generic.go:356] "Generic (PLEG): container finished" podID="a5bf63e0-9560-4894-84b6-1d5c399783a8" containerID="ec6b62b640e756be3e0f579ad22c628a2bf312749baa8c2f4c72e125426fa84f" exitCode=0 Oct 30 13:42:59.724181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:42:59.723750 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" event={"ID":"a5bf63e0-9560-4894-84b6-1d5c399783a8","Type":"ContainerDied","Data":"ec6b62b640e756be3e0f579ad22c628a2bf312749baa8c2f4c72e125426fa84f"} Oct 30 13:43:00.031699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:00.031598 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4"] Oct 30 13:43:00.037332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:00.037294 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh"] Oct 30 13:43:00.040931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:00.040910 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38f94b4-zv8t4"] Oct 30 13:43:00.044231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:00.044205 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb5ea26-2mxrh"] Oct 30 13:43:00.081582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:00.081546 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:00.385062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:00.385033 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:00.908900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:00.908866 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:43:01.045697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.045666 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-xks-ca\") pod \"a5bf63e0-9560-4894-84b6-1d5c399783a8\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " Oct 30 13:43:01.045877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.045743 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-openshift-ca\") pod \"a5bf63e0-9560-4894-84b6-1d5c399783a8\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " Oct 30 13:43:01.045990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.045971 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bf63e0-9560-4894-84b6-1d5c399783a8-kube-api-access-7zmgp\" (UniqueName: \"kubernetes.io/projected/a5bf63e0-9560-4894-84b6-1d5c399783a8-kube-api-access-7zmgp\") pod \"a5bf63e0-9560-4894-84b6-1d5c399783a8\" (UID: \"a5bf63e0-9560-4894-84b6-1d5c399783a8\") " Oct 30 13:43:01.046124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.046088 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-openshift-ca" pod "a5bf63e0-9560-4894-84b6-1d5c399783a8" (UID: "a5bf63e0-9560-4894-84b6-1d5c399783a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:01.046694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.046670 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:01.047892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.047867 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-xks-ca" pod "a5bf63e0-9560-4894-84b6-1d5c399783a8" (UID: "a5bf63e0-9560-4894-84b6-1d5c399783a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:01.048209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.048187 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bf63e0-9560-4894-84b6-1d5c399783a8-kube-api-access-7zmgp" pod "a5bf63e0-9560-4894-84b6-1d5c399783a8" (UID: "a5bf63e0-9560-4894-84b6-1d5c399783a8"). InnerVolumeSpecName "kube-api-access-7zmgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:01.147685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.147652 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zmgp\" (UniqueName: \"kubernetes.io/projected/a5bf63e0-9560-4894-84b6-1d5c399783a8-kube-api-access-7zmgp\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:01.147685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.147682 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bf63e0-9560-4894-84b6-1d5c399783a8-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:01.337572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.337492 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e80a744-135d-48f0-8f51-f7655b2e54a2" path="/var/lib/kubelet/pods/9e80a744-135d-48f0-8f51-f7655b2e54a2/volumes" Oct 30 13:43:01.338051 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.338036 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa9db65-bae4-4118-8eee-ec357fcee0a1" path="/var/lib/kubelet/pods/faa9db65-bae4-4118-8eee-ec357fcee0a1/volumes" Oct 30 13:43:01.743075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.742983 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" Oct 30 13:43:01.743075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.743011 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc" event={"ID":"a5bf63e0-9560-4894-84b6-1d5c399783a8","Type":"ContainerDied","Data":"2546c66fc69b5a74686101a0277c6104ef485d108b98d6af11cd4e9b81f74e23"} Oct 30 13:43:01.743075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:01.743066 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2546c66fc69b5a74686101a0277c6104ef485d108b98d6af11cd4e9b81f74e23" Oct 30 13:43:02.067603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:02.067563 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:04.419528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:04.419480 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:05.639064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:05.639019 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" podUID="feff1c90-5a44-4a9e-bef2-2992eeac9ee6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:43:05.639615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:05.639082 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" podUID="819b0f8d-71fd-4fed-ae5d-d619a030ffc5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:43:06.277615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.277571 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:06.683581 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.683555 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:43:06.736217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.735845 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:43:06.788063 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.788029 5049 generic.go:356] "Generic (PLEG): container finished" podID="819b0f8d-71fd-4fed-ae5d-d619a030ffc5" containerID="2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9" exitCode=0 Oct 30 13:43:06.788254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.788079 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" Oct 30 13:43:06.788254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.788115 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" event={"ID":"819b0f8d-71fd-4fed-ae5d-d619a030ffc5","Type":"ContainerDied","Data":"2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9"} Oct 30 13:43:06.788254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.788168 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f" event={"ID":"819b0f8d-71fd-4fed-ae5d-d619a030ffc5","Type":"ContainerDied","Data":"8515a9727c79e6f91dbfec5b39bffc058a075a6a250ed7aa86a3cdbe51acd123"} Oct 30 13:43:06.788254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.788186 5049 scope.go:122] "RemoveContainer" containerID="2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9" Oct 30 13:43:06.792946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.792917 5049 generic.go:356] "Generic (PLEG): container finished" podID="feff1c90-5a44-4a9e-bef2-2992eeac9ee6" containerID="727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2" exitCode=0 Oct 30 13:43:06.793120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.793011 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" Oct 30 13:43:06.793120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.793091 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" event={"ID":"feff1c90-5a44-4a9e-bef2-2992eeac9ee6","Type":"ContainerDied","Data":"727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2"} Oct 30 13:43:06.793260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.793136 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r" event={"ID":"feff1c90-5a44-4a9e-bef2-2992eeac9ee6","Type":"ContainerDied","Data":"3ea3467d79aa077f9d0b8d1d2560a2dc2a24c772082d3d5ec698402f47c0601a"} Oct 30 13:43:06.805818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.805795 5049 scope.go:122] "RemoveContainer" containerID="2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9" Oct 30 13:43:06.806116 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:06.806095 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9\": container with ID starting with 2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9 not found: ID does not exist" containerID="2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9" Oct 30 13:43:06.806231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.806127 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9"} err="failed to get container status \"2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9\": rpc error: code = NotFound desc = could not find container \"2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9\": container with ID starting with 2b0e7866eb4896929d5ecd7236a2b11620c69de09a2dddbdf8843ee1638ce0f9 not found: ID does not exist" Oct 30 13:43:06.806231 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.806150 5049 scope.go:122] "RemoveContainer" containerID="727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2" Oct 30 13:43:06.820949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.820916 5049 scope.go:122] "RemoveContainer" containerID="727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2" Oct 30 13:43:06.821266 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:06.821242 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2\": container with ID starting with 727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2 not found: ID does not exist" containerID="727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2" Oct 30 13:43:06.821379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.821274 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2"} err="failed to get container status \"727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2\": rpc error: code = NotFound desc = could not find container \"727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2\": container with ID starting with 727b3651be305189a6580c67ae8d5f71a32bf3b4389f85133dd69902760cc9e2 not found: ID does not exist" Oct 30 13:43:06.826870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.825898 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-plugins-config-volume\") pod \"819b0f8d-71fd-4fed-ae5d-d619a030ffc5\" (UID: \"819b0f8d-71fd-4fed-ae5d-d619a030ffc5\") " Oct 30 13:43:06.826870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.825961 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-kube-api-access-2gqbx\" (UniqueName: \"kubernetes.io/projected/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-kube-api-access-2gqbx\") pod \"feff1c90-5a44-4a9e-bef2-2992eeac9ee6\" (UID: \"feff1c90-5a44-4a9e-bef2-2992eeac9ee6\") " Oct 30 13:43:06.826870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.826031 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-kube-api-access-s64wm\" (UniqueName: \"kubernetes.io/projected/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-kube-api-access-s64wm\") pod \"819b0f8d-71fd-4fed-ae5d-d619a030ffc5\" (UID: \"819b0f8d-71fd-4fed-ae5d-d619a030ffc5\") " Oct 30 13:43:06.826870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.826110 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-plugins-config-volume\") pod \"feff1c90-5a44-4a9e-bef2-2992eeac9ee6\" (UID: \"feff1c90-5a44-4a9e-bef2-2992eeac9ee6\") " Oct 30 13:43:06.826870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.826262 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-plugins-config-volume" pod "819b0f8d-71fd-4fed-ae5d-d619a030ffc5" (UID: "819b0f8d-71fd-4fed-ae5d-d619a030ffc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:06.826870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.826841 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:06.827268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.827084 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-plugins-config-volume" pod "feff1c90-5a44-4a9e-bef2-2992eeac9ee6" (UID: "feff1c90-5a44-4a9e-bef2-2992eeac9ee6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:06.828994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.828968 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-kube-api-access-2gqbx" pod "feff1c90-5a44-4a9e-bef2-2992eeac9ee6" (UID: "feff1c90-5a44-4a9e-bef2-2992eeac9ee6"). InnerVolumeSpecName "kube-api-access-2gqbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:06.829981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.829959 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-kube-api-access-s64wm" pod "819b0f8d-71fd-4fed-ae5d-d619a030ffc5" (UID: "819b0f8d-71fd-4fed-ae5d-d619a030ffc5"). InnerVolumeSpecName "kube-api-access-s64wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:06.928079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.928045 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gqbx\" (UniqueName: \"kubernetes.io/projected/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-kube-api-access-2gqbx\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:06.928079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.928074 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s64wm\" (UniqueName: \"kubernetes.io/projected/819b0f8d-71fd-4fed-ae5d-d619a030ffc5-kube-api-access-s64wm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:06.928079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:06.928084 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feff1c90-5a44-4a9e-bef2-2992eeac9ee6-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:07.133238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:07.133209 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f"] Oct 30 13:43:07.142893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:07.142868 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c47903-658c46f75f-hw29f"] Oct 30 13:43:07.166388 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:07.166361 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r"] Oct 30 13:43:07.171228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:07.171198 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c47903-787cb5dfd6-psk6r"] Oct 30 13:43:07.338927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:07.338891 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819b0f8d-71fd-4fed-ae5d-d619a030ffc5" path="/var/lib/kubelet/pods/819b0f8d-71fd-4fed-ae5d-d619a030ffc5/volumes" Oct 30 13:43:07.339567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:07.339545 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feff1c90-5a44-4a9e-bef2-2992eeac9ee6" path="/var/lib/kubelet/pods/feff1c90-5a44-4a9e-bef2-2992eeac9ee6/volumes" Oct 30 13:43:08.065150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:08.064468 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:10.483976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:10.483904 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:12.372471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:12.372440 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:14.122485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:14.122385 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:15.905146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:15.905064 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:17.723957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:17.723912 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:19.289145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:19.289115 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:19.869331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:19.869284 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:19.961110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:19.961076 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn"] Oct 30 13:43:19.970526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:19.970497 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c47903-rhgrn"] Oct 30 13:43:20.033119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:20.031788 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:20.512447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:20.512413 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:21.337037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:21.337001 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47807a8c-a577-4e49-98b4-25b97684ec53" path="/var/lib/kubelet/pods/47807a8c-a577-4e49-98b4-25b97684ec53/volumes" Oct 30 13:43:21.930789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:21.930738 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:23.800467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:23.800419 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:25.523423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:25.523389 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:25.599413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:25.599377 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt"] Oct 30 13:43:25.599637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:25.599610 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" podUID="70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e" containerName="payload-processing" containerID="cri-o://7e6073c6f385d6669e77512999d9f656d4406d3bae239b87005b970d01f6c1ad" gracePeriod=30 Oct 30 13:43:25.630423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:25.630393 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk"] Oct 30 13:43:25.630691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:25.630660 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" podUID="2cf4f011-ec24-40c7-a0ab-2bb59f48ad36" containerName="payload-pre-processing" containerID="cri-o://f66176d978888c8b1242d6c79f39bc8dbab7ae071696f22c8c252987868c2a14" gracePeriod=30 Oct 30 13:43:25.745705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:25.745673 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:25.933358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:25.920726 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:26.097287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.096987 5049 generic.go:356] "Generic (PLEG): container finished" podID="70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e" containerID="7e6073c6f385d6669e77512999d9f656d4406d3bae239b87005b970d01f6c1ad" exitCode=0 Oct 30 13:43:26.097287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.097147 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" event={"ID":"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e","Type":"ContainerDied","Data":"7e6073c6f385d6669e77512999d9f656d4406d3bae239b87005b970d01f6c1ad"} Oct 30 13:43:26.117790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.117761 5049 generic.go:356] "Generic (PLEG): container finished" podID="2cf4f011-ec24-40c7-a0ab-2bb59f48ad36" containerID="f66176d978888c8b1242d6c79f39bc8dbab7ae071696f22c8c252987868c2a14" exitCode=0 Oct 30 13:43:26.118100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.118076 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" event={"ID":"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36","Type":"ContainerDied","Data":"f66176d978888c8b1242d6c79f39bc8dbab7ae071696f22c8c252987868c2a14"} Oct 30 13:43:26.153627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.153599 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:43:26.156836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.156813 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:43:26.241002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.240956 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-kube-api-access-nnwbc\" (UniqueName: \"kubernetes.io/projected/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-kube-api-access-nnwbc\") pod \"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e\" (UID: \"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e\") " Oct 30 13:43:26.241208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.241052 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-plugins-config-volume\") pod \"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36\" (UID: \"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36\") " Oct 30 13:43:26.241208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.241151 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-plugins-config-volume\") pod \"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e\" (UID: \"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e\") " Oct 30 13:43:26.241357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.241265 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-kube-api-access-f2gz2\" (UniqueName: \"kubernetes.io/projected/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-kube-api-access-f2gz2\") pod \"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36\" (UID: \"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36\") " Oct 30 13:43:26.241911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.241868 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-plugins-config-volume" pod "70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e" (UID: "70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:26.242056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.242004 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:26.243037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.243016 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-plugins-config-volume" pod "2cf4f011-ec24-40c7-a0ab-2bb59f48ad36" (UID: "2cf4f011-ec24-40c7-a0ab-2bb59f48ad36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:26.244595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.244562 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-kube-api-access-f2gz2" pod "2cf4f011-ec24-40c7-a0ab-2bb59f48ad36" (UID: "2cf4f011-ec24-40c7-a0ab-2bb59f48ad36"). InnerVolumeSpecName "kube-api-access-f2gz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:26.246600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.246579 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-kube-api-access-nnwbc" pod "70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e" (UID: "70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e"). InnerVolumeSpecName "kube-api-access-nnwbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:26.343188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.343150 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnwbc\" (UniqueName: \"kubernetes.io/projected/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e-kube-api-access-nnwbc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:26.343188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.343183 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:26.343188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.343195 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2gz2\" (UniqueName: \"kubernetes.io/projected/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36-kube-api-access-f2gz2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:26.352789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.352711 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:26.437997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.437967 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8"] Oct 30 13:43:26.441690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.441662 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e3ebf5-tv8h8"] Oct 30 13:43:26.543344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.543259 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:26.677669 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:26.677573 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:43:26.973536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:26.973454 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:27.127548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.127520 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" Oct 30 13:43:27.127733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.127526 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt" event={"ID":"70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e","Type":"ContainerDied","Data":"b0cf265fa98aef7e3446173e0062edfde643668cd6952ae4ec5ed872d5a14bb2"} Oct 30 13:43:27.127733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.127663 5049 scope.go:122] "RemoveContainer" containerID="7e6073c6f385d6669e77512999d9f656d4406d3bae239b87005b970d01f6c1ad" Oct 30 13:43:27.129128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.129107 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" Oct 30 13:43:27.129264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.129134 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk" event={"ID":"2cf4f011-ec24-40c7-a0ab-2bb59f48ad36","Type":"ContainerDied","Data":"bbfb25959458369cebef840a86a6a31c7b6debca90cedf0a6d980ef018529e66"} Oct 30 13:43:27.140410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.140353 5049 scope.go:122] "RemoveContainer" containerID="f66176d978888c8b1242d6c79f39bc8dbab7ae071696f22c8c252987868c2a14" Oct 30 13:43:27.178108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.178082 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk"] Oct 30 13:43:27.187852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.187831 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e3ebf5-6cc6454c6c-l4zkk"] Oct 30 13:43:27.211216 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.211189 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt"] Oct 30 13:43:27.215021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.214996 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e3ebf5-6dfcdd96-r56dt"] Oct 30 13:43:27.340466 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.340437 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf4f011-ec24-40c7-a0ab-2bb59f48ad36" path="/var/lib/kubelet/pods/2cf4f011-ec24-40c7-a0ab-2bb59f48ad36/volumes" Oct 30 13:43:27.340924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.340911 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e" path="/var/lib/kubelet/pods/70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e/volumes" Oct 30 13:43:27.341350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.341312 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73504fcb-46a4-451f-ade1-c9b5edfa0940" path="/var/lib/kubelet/pods/73504fcb-46a4-451f-ade1-c9b5edfa0940/volumes" Oct 30 13:43:27.967237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:27.967205 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:28.124802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.124764 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:28.312376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.298544 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j"] Oct 30 13:43:28.312376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.298784 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" containerID="cri-o://05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0" gracePeriod=30 Oct 30 13:43:28.430079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.430037 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:28.654390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.654297 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:28.834026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.833987 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j"] Oct 30 13:43:28.834767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834750 5049 cpu_manager.go:414] "containerMap: removing container" podUID="70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e" containerName="payload-processing" Oct 30 13:43:28.834822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834769 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e" containerName="payload-processing" Oct 30 13:43:28.834822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834808 5049 cpu_manager.go:414] "containerMap: removing container" podUID="feff1c90-5a44-4a9e-bef2-2992eeac9ee6" containerName="payload-processing" Oct 30 13:43:28.834822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834814 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feff1c90-5a44-4a9e-bef2-2992eeac9ee6" containerName="payload-processing" Oct 30 13:43:28.834930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834829 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2cf4f011-ec24-40c7-a0ab-2bb59f48ad36" containerName="payload-pre-processing" Oct 30 13:43:28.834930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834836 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf4f011-ec24-40c7-a0ab-2bb59f48ad36" containerName="payload-pre-processing" Oct 30 13:43:28.834930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834849 5049 cpu_manager.go:414] "containerMap: removing container" podUID="819b0f8d-71fd-4fed-ae5d-d619a030ffc5" containerName="payload-pre-processing" Oct 30 13:43:28.834930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834854 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819b0f8d-71fd-4fed-ae5d-d619a030ffc5" containerName="payload-pre-processing" Oct 30 13:43:28.834930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834880 5049 cpu_manager.go:414] "containerMap: removing container" podUID="a5bf63e0-9560-4894-84b6-1d5c399783a8" containerName="revoke-keys" Oct 30 13:43:28.834930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834885 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bf63e0-9560-4894-84b6-1d5c399783a8" containerName="revoke-keys" Oct 30 13:43:28.835136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.834990 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bf63e0-9560-4894-84b6-1d5c399783a8" containerName="revoke-keys" Oct 30 13:43:28.835136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.835001 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="70551eed-5a9f-4fb9-9b5d-b8f0fbb0277e" containerName="payload-processing" Oct 30 13:43:28.835136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.835010 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="feff1c90-5a44-4a9e-bef2-2992eeac9ee6" containerName="payload-processing" Oct 30 13:43:28.835136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.835017 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf4f011-ec24-40c7-a0ab-2bb59f48ad36" containerName="payload-pre-processing" Oct 30 13:43:28.835136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.835024 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="819b0f8d-71fd-4fed-ae5d-d619a030ffc5" containerName="payload-pre-processing" Oct 30 13:43:28.840236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.840218 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:28.862447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.862420 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j"] Oct 30 13:43:28.864601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.864577 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:43:28.864777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.864694 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:43:28.888489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.888454 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:28.888667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.888633 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4c4f\" (UniqueName: \"kubernetes.io/projected/89e0892b-70a8-4303-9e89-a62b1dce067d-kube-api-access-w4c4f\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:28.888757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.888738 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:28.990509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.990415 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:28.990983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.990540 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4c4f\" (UniqueName: \"kubernetes.io/projected/89e0892b-70a8-4303-9e89-a62b1dce067d-kube-api-access-w4c4f\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:28.990983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.990591 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:28.991189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.991167 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:28.993180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:28.993153 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:29.013476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:29.013448 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4c4f\" (UniqueName: \"kubernetes.io/projected/89e0892b-70a8-4303-9e89-a62b1dce067d-kube-api-access-w4c4f\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:29.154960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:29.154924 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:29.466736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:29.466465 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j"] Oct 30 13:43:29.469737 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:43:29.469705 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e0892b_70a8_4303_9e89_a62b1dce067d.slice/crio-0ca57fe1b8d4a0f290fabd9c7fbf2269d3c42dfd4220084fd792da9fcb134488 WatchSource:0}: Error finding container 0ca57fe1b8d4a0f290fabd9c7fbf2269d3c42dfd4220084fd792da9fcb134488: Status 404 returned error can't find the container with id 0ca57fe1b8d4a0f290fabd9c7fbf2269d3c42dfd4220084fd792da9fcb134488 Oct 30 13:43:30.066614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:30.066574 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:43:30.067029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:30.066638 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:43:30.182013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:30.181913 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" event={"ID":"89e0892b-70a8-4303-9e89-a62b1dce067d","Type":"ContainerStarted","Data":"48780cf2f95369208e760ede4454241451dc7bbb0362cc92caaea0ae0c027f97"} Oct 30 13:43:30.182013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:30.181976 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" event={"ID":"89e0892b-70a8-4303-9e89-a62b1dce067d","Type":"ContainerStarted","Data":"0ca57fe1b8d4a0f290fabd9c7fbf2269d3c42dfd4220084fd792da9fcb134488"} Oct 30 13:43:30.223301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:30.223233 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" podStartSLOduration=2.223212982 podStartE2EDuration="2.223212982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:30.219143333 +0000 UTC m=+3117.034337756" watchObservedRunningTime="2026-10-30 13:43:30.223212982 +0000 UTC m=+3117.038407406" Oct 30 13:43:30.352078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:30.352039 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:30.604010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:30.603962 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:31.192022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:31.191986 5049 generic.go:356] "Generic (PLEG): container finished" podID="89e0892b-70a8-4303-9e89-a62b1dce067d" containerID="48780cf2f95369208e760ede4454241451dc7bbb0362cc92caaea0ae0c027f97" exitCode=0 Oct 30 13:43:31.192476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:31.192069 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" event={"ID":"89e0892b-70a8-4303-9e89-a62b1dce067d","Type":"ContainerDied","Data":"48780cf2f95369208e760ede4454241451dc7bbb0362cc92caaea0ae0c027f97"} Oct 30 13:43:32.392462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.392435 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:32.393502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.393468 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:32.439768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.439734 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-openshift-ca\") pod \"89e0892b-70a8-4303-9e89-a62b1dce067d\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " Oct 30 13:43:32.439955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.439837 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e0892b-70a8-4303-9e89-a62b1dce067d-kube-api-access-w4c4f\" (UniqueName: \"kubernetes.io/projected/89e0892b-70a8-4303-9e89-a62b1dce067d-kube-api-access-w4c4f\") pod \"89e0892b-70a8-4303-9e89-a62b1dce067d\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " Oct 30 13:43:32.440026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.439993 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-xks-ca\") pod \"89e0892b-70a8-4303-9e89-a62b1dce067d\" (UID: \"89e0892b-70a8-4303-9e89-a62b1dce067d\") " Oct 30 13:43:32.440190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.440162 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-openshift-ca" pod "89e0892b-70a8-4303-9e89-a62b1dce067d" (UID: "89e0892b-70a8-4303-9e89-a62b1dce067d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:32.440590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.440563 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:32.442575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.442547 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-xks-ca" pod "89e0892b-70a8-4303-9e89-a62b1dce067d" (UID: "89e0892b-70a8-4303-9e89-a62b1dce067d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:32.455945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.455876 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e0892b-70a8-4303-9e89-a62b1dce067d-kube-api-access-w4c4f" pod "89e0892b-70a8-4303-9e89-a62b1dce067d" (UID: "89e0892b-70a8-4303-9e89-a62b1dce067d"). InnerVolumeSpecName "kube-api-access-w4c4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:32.488055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.488016 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6"] Oct 30 13:43:32.488273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.488247 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" podUID="647f2071-e66f-4f37-bff7-6bbfb60dd5c1" containerName="payload-processing" containerID="cri-o://98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5" gracePeriod=30 Oct 30 13:43:32.517048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.514778 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5"] Oct 30 13:43:32.517048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.515052 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" podUID="c5a91a94-c1e6-4afc-8576-631b1361582f" containerName="payload-pre-processing" containerID="cri-o://30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf" gracePeriod=30 Oct 30 13:43:32.541983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.541944 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4c4f\" (UniqueName: \"kubernetes.io/projected/89e0892b-70a8-4303-9e89-a62b1dce067d-kube-api-access-w4c4f\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:32.541983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.541985 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e0892b-70a8-4303-9e89-a62b1dce067d-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:32.623903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.623868 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:32.795585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.795541 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:32.812839 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.812811 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:43:32.982393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:32.980553 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:43:33.039379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.039310 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a91a94-c1e6-4afc-8576-631b1361582f-kube-api-access-692z5\" (UniqueName: \"kubernetes.io/projected/c5a91a94-c1e6-4afc-8576-631b1361582f-kube-api-access-692z5\") pod \"c5a91a94-c1e6-4afc-8576-631b1361582f\" (UID: \"c5a91a94-c1e6-4afc-8576-631b1361582f\") " Oct 30 13:43:33.039580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.039501 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a91a94-c1e6-4afc-8576-631b1361582f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a91a94-c1e6-4afc-8576-631b1361582f-plugins-config-volume\") pod \"c5a91a94-c1e6-4afc-8576-631b1361582f\" (UID: \"c5a91a94-c1e6-4afc-8576-631b1361582f\") " Oct 30 13:43:33.040074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.040044 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a91a94-c1e6-4afc-8576-631b1361582f-plugins-config-volume" pod "c5a91a94-c1e6-4afc-8576-631b1361582f" (UID: "c5a91a94-c1e6-4afc-8576-631b1361582f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:33.042718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.042691 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a91a94-c1e6-4afc-8576-631b1361582f-kube-api-access-692z5" pod "c5a91a94-c1e6-4afc-8576-631b1361582f" (UID: "c5a91a94-c1e6-4afc-8576-631b1361582f"). InnerVolumeSpecName "kube-api-access-692z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:33.141007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.140974 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-plugins-config-volume\") pod \"647f2071-e66f-4f37-bff7-6bbfb60dd5c1\" (UID: \"647f2071-e66f-4f37-bff7-6bbfb60dd5c1\") " Oct 30 13:43:33.141223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.141053 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-kube-api-access-s4vcl\" (UniqueName: \"kubernetes.io/projected/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-kube-api-access-s4vcl\") pod \"647f2071-e66f-4f37-bff7-6bbfb60dd5c1\" (UID: \"647f2071-e66f-4f37-bff7-6bbfb60dd5c1\") " Oct 30 13:43:33.141456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.141432 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-plugins-config-volume" pod "647f2071-e66f-4f37-bff7-6bbfb60dd5c1" (UID: "647f2071-e66f-4f37-bff7-6bbfb60dd5c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:33.141691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.141673 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.141751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.141694 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-692z5\" (UniqueName: \"kubernetes.io/projected/c5a91a94-c1e6-4afc-8576-631b1361582f-kube-api-access-692z5\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.141751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.141703 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a91a94-c1e6-4afc-8576-631b1361582f-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.143192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.143165 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-kube-api-access-s4vcl" pod "647f2071-e66f-4f37-bff7-6bbfb60dd5c1" (UID: "647f2071-e66f-4f37-bff7-6bbfb60dd5c1"). InnerVolumeSpecName "kube-api-access-s4vcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:33.214272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.214238 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" Oct 30 13:43:33.214492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.214241 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j" event={"ID":"89e0892b-70a8-4303-9e89-a62b1dce067d","Type":"ContainerDied","Data":"0ca57fe1b8d4a0f290fabd9c7fbf2269d3c42dfd4220084fd792da9fcb134488"} Oct 30 13:43:33.214492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.214418 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca57fe1b8d4a0f290fabd9c7fbf2269d3c42dfd4220084fd792da9fcb134488" Oct 30 13:43:33.215863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.215833 5049 generic.go:356] "Generic (PLEG): container finished" podID="c5a91a94-c1e6-4afc-8576-631b1361582f" containerID="30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf" exitCode=0 Oct 30 13:43:33.216028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.215880 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" Oct 30 13:43:33.216028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.215923 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" event={"ID":"c5a91a94-c1e6-4afc-8576-631b1361582f","Type":"ContainerDied","Data":"30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf"} Oct 30 13:43:33.216028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.215992 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5" event={"ID":"c5a91a94-c1e6-4afc-8576-631b1361582f","Type":"ContainerDied","Data":"567a4064a065d80f15dd0c055581954b2b006750925335bc75271eddfb5a589b"} Oct 30 13:43:33.216028 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.216026 5049 scope.go:122] "RemoveContainer" containerID="30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf" Oct 30 13:43:33.217990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.217955 5049 generic.go:356] "Generic (PLEG): container finished" podID="647f2071-e66f-4f37-bff7-6bbfb60dd5c1" containerID="98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5" exitCode=0 Oct 30 13:43:33.218119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.218027 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" event={"ID":"647f2071-e66f-4f37-bff7-6bbfb60dd5c1","Type":"ContainerDied","Data":"98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5"} Oct 30 13:43:33.218119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.218068 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" event={"ID":"647f2071-e66f-4f37-bff7-6bbfb60dd5c1","Type":"ContainerDied","Data":"8e627e9ff2e0cc582ec57a3e973b2128ed7148c8a47a6e0d3d845a9ae70d516e"} Oct 30 13:43:33.218243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.218128 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6" Oct 30 13:43:33.232987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.232952 5049 scope.go:122] "RemoveContainer" containerID="30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf" Oct 30 13:43:33.233260 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:33.233236 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf\": container with ID starting with 30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf not found: ID does not exist" containerID="30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf" Oct 30 13:43:33.233370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.233272 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf"} err="failed to get container status \"30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf\": rpc error: code = NotFound desc = could not find container \"30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf\": container with ID starting with 30ad2d4fc44d5c0cebb851f8860d6c69c63bc8313aab9401c2ecba5a0f8c38cf not found: ID does not exist" Oct 30 13:43:33.233370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.233296 5049 scope.go:122] "RemoveContainer" containerID="98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5" Oct 30 13:43:33.243301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.243271 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4vcl\" (UniqueName: \"kubernetes.io/projected/647f2071-e66f-4f37-bff7-6bbfb60dd5c1-kube-api-access-s4vcl\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.245438 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.245418 5049 scope.go:122] "RemoveContainer" containerID="98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5" Oct 30 13:43:33.245741 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:33.245721 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5\": container with ID starting with 98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5 not found: ID does not exist" containerID="98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5" Oct 30 13:43:33.245825 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.245748 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5"} err="failed to get container status \"98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5\": rpc error: code = NotFound desc = could not find container \"98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5\": container with ID starting with 98d550b4476c5d3ff69cfb5bd05610f1057e92957e1b8daa2c62abe42aab70b5 not found: ID does not exist" Oct 30 13:43:33.326400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.326369 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5"] Oct 30 13:43:33.356678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.356652 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-65bc587c67-dzlr5"] Oct 30 13:43:33.404121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.404086 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6"] Oct 30 13:43:33.409353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.409308 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-56cfb45d5b-8wqm6"] Oct 30 13:43:33.417482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.417454 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:33.468424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.466555 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc"] Oct 30 13:43:33.470378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.470352 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-30c004-mgxjc"] Oct 30 13:43:33.605078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.605052 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:43:33.649083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.649052 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:33.669571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.669538 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27db492b-5ede-4ac7-99e5-06c5bc19051f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27db492b-5ede-4ac7-99e5-06c5bc19051f-istiod-ca-cert\") pod \"27db492b-5ede-4ac7-99e5-06c5bc19051f\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " Oct 30 13:43:33.669773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.669594 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-podinfo\") pod \"27db492b-5ede-4ac7-99e5-06c5bc19051f\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " Oct 30 13:43:33.669773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.669697 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-token\" (UniqueName: \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-token\") pod \"27db492b-5ede-4ac7-99e5-06c5bc19051f\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " Oct 30 13:43:33.669773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.669742 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-credential-socket\") pod \"27db492b-5ede-4ac7-99e5-06c5bc19051f\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " Oct 30 13:43:33.669962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.669781 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-data\") pod \"27db492b-5ede-4ac7-99e5-06c5bc19051f\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " Oct 30 13:43:33.669962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.669857 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-certs\") pod \"27db492b-5ede-4ac7-99e5-06c5bc19051f\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " Oct 30 13:43:33.669962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.669944 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-socket\") pod \"27db492b-5ede-4ac7-99e5-06c5bc19051f\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " Oct 30 13:43:33.670136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.670021 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-envoy\") pod \"27db492b-5ede-4ac7-99e5-06c5bc19051f\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " Oct 30 13:43:33.670136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.670085 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-kube-api-access-fxz8z\" (UniqueName: \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-kube-api-access-fxz8z\") pod \"27db492b-5ede-4ac7-99e5-06c5bc19051f\" (UID: \"27db492b-5ede-4ac7-99e5-06c5bc19051f\") " Oct 30 13:43:33.671360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.671295 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-credential-socket" pod "27db492b-5ede-4ac7-99e5-06c5bc19051f" (UID: "27db492b-5ede-4ac7-99e5-06c5bc19051f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:33.671638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.671568 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-data" pod "27db492b-5ede-4ac7-99e5-06c5bc19051f" (UID: "27db492b-5ede-4ac7-99e5-06c5bc19051f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:33.671760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.671663 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-certs" pod "27db492b-5ede-4ac7-99e5-06c5bc19051f" (UID: "27db492b-5ede-4ac7-99e5-06c5bc19051f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:33.672037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.672003 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-socket" pod "27db492b-5ede-4ac7-99e5-06c5bc19051f" (UID: "27db492b-5ede-4ac7-99e5-06c5bc19051f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:33.672139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.672033 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27db492b-5ede-4ac7-99e5-06c5bc19051f-istiod-ca-cert" pod "27db492b-5ede-4ac7-99e5-06c5bc19051f" (UID: "27db492b-5ede-4ac7-99e5-06c5bc19051f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:33.683552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.683519 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-kube-api-access-fxz8z" pod "27db492b-5ede-4ac7-99e5-06c5bc19051f" (UID: "27db492b-5ede-4ac7-99e5-06c5bc19051f"). InnerVolumeSpecName "kube-api-access-fxz8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:33.683695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.683649 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-podinfo" pod "27db492b-5ede-4ac7-99e5-06c5bc19051f" (UID: "27db492b-5ede-4ac7-99e5-06c5bc19051f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:43:33.683764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.683737 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-token" pod "27db492b-5ede-4ac7-99e5-06c5bc19051f" (UID: "27db492b-5ede-4ac7-99e5-06c5bc19051f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:33.685180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.685146 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-envoy" pod "27db492b-5ede-4ac7-99e5-06c5bc19051f" (UID: "27db492b-5ede-4ac7-99e5-06c5bc19051f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:33.771936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.771852 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.771936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.771886 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.771936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.771896 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxz8z\" (UniqueName: \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-kube-api-access-fxz8z\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.771936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.771906 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27db492b-5ede-4ac7-99e5-06c5bc19051f-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.771936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.771915 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.771936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.771923 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.771936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.771930 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.771936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.771938 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:33.772309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:33.771946 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27db492b-5ede-4ac7-99e5-06c5bc19051f-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:34.035061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.034980 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:34.231639 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.231597 5049 generic.go:356] "Generic (PLEG): container finished" podID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerID="05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0" exitCode=0 Oct 30 13:43:34.231813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.231651 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" Oct 30 13:43:34.231813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.231680 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" event={"ID":"27db492b-5ede-4ac7-99e5-06c5bc19051f","Type":"ContainerDied","Data":"05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0"} Oct 30 13:43:34.231813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.231741 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j" event={"ID":"27db492b-5ede-4ac7-99e5-06c5bc19051f","Type":"ContainerDied","Data":"9c760e63b35952f2dd0bbd28884b6db4846ddb787dcc1a2506a8e35321ad8889"} Oct 30 13:43:34.231813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.231765 5049 scope.go:122] "RemoveContainer" containerID="05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0" Oct 30 13:43:34.247774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.247756 5049 scope.go:122] "RemoveContainer" containerID="8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b" Oct 30 13:43:34.259214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.259196 5049 scope.go:122] "RemoveContainer" containerID="05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0" Oct 30 13:43:34.259522 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:34.259502 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0\": container with ID starting with 05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0 not found: ID does not exist" containerID="05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0" Oct 30 13:43:34.259589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.259532 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0"} err="failed to get container status \"05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0\": rpc error: code = NotFound desc = could not find container \"05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0\": container with ID starting with 05fbc9ae1d04be9dc0780798423485b2b26d18fdce850ac888660cf49d9c29a0 not found: ID does not exist" Oct 30 13:43:34.259589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.259550 5049 scope.go:122] "RemoveContainer" containerID="8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b" Oct 30 13:43:34.259820 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:34.259801 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b\": container with ID starting with 8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b not found: ID does not exist" containerID="8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b" Oct 30 13:43:34.259872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.259831 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b"} err="failed to get container status \"8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b\": rpc error: code = NotFound desc = could not find container \"8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b\": container with ID starting with 8ebf439aa8f54c257d02a6d5c53a6b920f46adc8701a957a0432e28c8af08c1b not found: ID does not exist" Oct 30 13:43:34.289611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.289534 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j"] Oct 30 13:43:34.293004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.292977 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c47903-openshift-default-86bfbcd5f7-x2g6j"] Oct 30 13:43:34.938114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:34.938080 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:35.131658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.131605 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:35.285844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.285750 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl"] Oct 30 13:43:35.286062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.286033 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" containerID="cri-o://a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1" gracePeriod=30 Oct 30 13:43:35.338390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.338356 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" path="/var/lib/kubelet/pods/27db492b-5ede-4ac7-99e5-06c5bc19051f/volumes" Oct 30 13:43:35.338976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.338950 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647f2071-e66f-4f37-bff7-6bbfb60dd5c1" path="/var/lib/kubelet/pods/647f2071-e66f-4f37-bff7-6bbfb60dd5c1/volumes" Oct 30 13:43:35.339450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.339432 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662b2ff1-7704-4fd9-a418-c165b91f285c" path="/var/lib/kubelet/pods/662b2ff1-7704-4fd9-a418-c165b91f285c/volumes" Oct 30 13:43:35.340223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.340144 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a91a94-c1e6-4afc-8576-631b1361582f" path="/var/lib/kubelet/pods/c5a91a94-c1e6-4afc-8576-631b1361582f/volumes" Oct 30 13:43:35.418673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.409384 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:35.679185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.679154 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:35.869107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869067 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9"] Oct 30 13:43:35.869786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869771 5049 cpu_manager.go:414] "containerMap: removing container" podUID="647f2071-e66f-4f37-bff7-6bbfb60dd5c1" containerName="payload-processing" Oct 30 13:43:35.869841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869787 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647f2071-e66f-4f37-bff7-6bbfb60dd5c1" containerName="payload-processing" Oct 30 13:43:35.869841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869810 5049 cpu_manager.go:414] "containerMap: removing container" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" Oct 30 13:43:35.869841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869815 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" Oct 30 13:43:35.869841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869825 5049 cpu_manager.go:414] "containerMap: removing container" podUID="89e0892b-70a8-4303-9e89-a62b1dce067d" containerName="revoke-keys" Oct 30 13:43:35.869841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869830 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e0892b-70a8-4303-9e89-a62b1dce067d" containerName="revoke-keys" Oct 30 13:43:35.870003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869854 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c5a91a94-c1e6-4afc-8576-631b1361582f" containerName="payload-pre-processing" Oct 30 13:43:35.870003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869859 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a91a94-c1e6-4afc-8576-631b1361582f" containerName="payload-pre-processing" Oct 30 13:43:35.870003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869872 5049 cpu_manager.go:414] "containerMap: removing container" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" Oct 30 13:43:35.870003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869877 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" Oct 30 13:43:35.870003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869981 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" Oct 30 13:43:35.870003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869988 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e0892b-70a8-4303-9e89-a62b1dce067d" containerName="revoke-keys" Oct 30 13:43:35.870003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.869999 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a91a94-c1e6-4afc-8576-631b1361582f" containerName="payload-pre-processing" Oct 30 13:43:35.870309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.870008 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="27db492b-5ede-4ac7-99e5-06c5bc19051f" containerName="istio-proxy" Oct 30 13:43:35.870309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.870016 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="647f2071-e66f-4f37-bff7-6bbfb60dd5c1" containerName="payload-processing" Oct 30 13:43:35.875370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.875347 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:35.895931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.895903 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:43:35.896050 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.895933 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:43:35.898921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.898900 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:35.899012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.898965 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtqm\" (UniqueName: \"kubernetes.io/projected/2f5bdbc6-d487-431f-a741-b47a041b1ca2-kube-api-access-kjtqm\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:35.899138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.899119 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:35.903883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:35.903862 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9"] Oct 30 13:43:36.000711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:36.000628 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:36.000711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:36.000696 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtqm\" (UniqueName: \"kubernetes.io/projected/2f5bdbc6-d487-431f-a741-b47a041b1ca2-kube-api-access-kjtqm\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:36.001118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:36.000739 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:36.001295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:36.001272 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:36.003120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:36.003099 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:36.022582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:36.022553 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtqm\" (UniqueName: \"kubernetes.io/projected/2f5bdbc6-d487-431f-a741-b47a041b1ca2-kube-api-access-kjtqm\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:36.188847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:36.188814 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:36.505439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:36.505299 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9"] Oct 30 13:43:36.510304 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:43:36.510271 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5bdbc6_d487_431f_a741_b47a041b1ca2.slice/crio-d20877a4885e5542e9f074be45e309b0f696dc45b5ee4d2a12ec9d3f4b2163d2 WatchSource:0}: Error finding container d20877a4885e5542e9f074be45e309b0f696dc45b5ee4d2a12ec9d3f4b2163d2: Status 404 returned error can't find the container with id d20877a4885e5542e9f074be45e309b0f696dc45b5ee4d2a12ec9d3f4b2163d2 Oct 30 13:43:36.813816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:36.813779 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:37.285227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.285133 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" event={"ID":"2f5bdbc6-d487-431f-a741-b47a041b1ca2","Type":"ContainerStarted","Data":"f4dffd41d091dfa01e3846e63eb850b90104396fb574e7564d8d1d0a29b38231"} Oct 30 13:43:37.285227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.285212 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" event={"ID":"2f5bdbc6-d487-431f-a741-b47a041b1ca2","Type":"ContainerStarted","Data":"d20877a4885e5542e9f074be45e309b0f696dc45b5ee4d2a12ec9d3f4b2163d2"} Oct 30 13:43:37.324392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.324361 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:43:37.324567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.324412 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:43:37.327703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.327658 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" podStartSLOduration=2.327644875 podStartE2EDuration="2.327644875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:37.325495325 +0000 UTC m=+3124.140689749" watchObservedRunningTime="2026-10-30 13:43:37.327644875 +0000 UTC m=+3124.142839297" Oct 30 13:43:37.411518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.411486 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:37.618989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.618956 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls"] Oct 30 13:43:37.619234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.619208 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" podUID="56ffe634-75ee-4ed1-abea-7dda3aad4a0a" containerName="payload-processing" containerID="cri-o://1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d" gracePeriod=30 Oct 30 13:43:37.734787 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.734746 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq"] Oct 30 13:43:37.735020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.734994 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" podUID="2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd" containerName="payload-pre-processing" containerID="cri-o://ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034" gracePeriod=30 Oct 30 13:43:37.790027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.790000 5049 scope.go:122] "RemoveContainer" containerID="f015dc2b01d074d1c9873d2c78017744f1b42715c9f5833c7738c3f1a02aa9e4" Oct 30 13:43:37.793151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.793127 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:37.967962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.967833 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:37.968715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:37.968688 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:43:38.087201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.043769 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-kube-api-access-wff25\" (UniqueName: \"kubernetes.io/projected/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-kube-api-access-wff25\") pod \"56ffe634-75ee-4ed1-abea-7dda3aad4a0a\" (UID: \"56ffe634-75ee-4ed1-abea-7dda3aad4a0a\") " Oct 30 13:43:38.087201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.044059 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-plugins-config-volume\") pod \"56ffe634-75ee-4ed1-abea-7dda3aad4a0a\" (UID: \"56ffe634-75ee-4ed1-abea-7dda3aad4a0a\") " Oct 30 13:43:38.087914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.087856 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-plugins-config-volume" pod "56ffe634-75ee-4ed1-abea-7dda3aad4a0a" (UID: "56ffe634-75ee-4ed1-abea-7dda3aad4a0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:38.147423 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.145775 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:38.161901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.161695 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-kube-api-access-wff25" pod "56ffe634-75ee-4ed1-abea-7dda3aad4a0a" (UID: "56ffe634-75ee-4ed1-abea-7dda3aad4a0a"). InnerVolumeSpecName "kube-api-access-wff25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:38.233942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.233913 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:43:38.246222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.246184 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-plugins-config-volume\") pod \"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd\" (UID: \"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd\") " Oct 30 13:43:38.246390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.246310 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-kube-api-access-cvl2n\" (UniqueName: \"kubernetes.io/projected/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-kube-api-access-cvl2n\") pod \"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd\" (UID: \"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd\") " Oct 30 13:43:38.246568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.246540 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-plugins-config-volume" pod "2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd" (UID: "2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:38.246850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.246835 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wff25\" (UniqueName: \"kubernetes.io/projected/56ffe634-75ee-4ed1-abea-7dda3aad4a0a-kube-api-access-wff25\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:38.246919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.246853 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:38.248510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.248487 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-kube-api-access-cvl2n" pod "2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd" (UID: "2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd"). InnerVolumeSpecName "kube-api-access-cvl2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:38.299255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.299218 5049 generic.go:356] "Generic (PLEG): container finished" podID="56ffe634-75ee-4ed1-abea-7dda3aad4a0a" containerID="1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d" exitCode=0 Oct 30 13:43:38.299801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.299282 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" Oct 30 13:43:38.299801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.299308 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" event={"ID":"56ffe634-75ee-4ed1-abea-7dda3aad4a0a","Type":"ContainerDied","Data":"1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d"} Oct 30 13:43:38.299801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.299404 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls" event={"ID":"56ffe634-75ee-4ed1-abea-7dda3aad4a0a","Type":"ContainerDied","Data":"8f7e9dfb059367506666379d24bf2df49edc581ff93ef92929a1d867ec61473c"} Oct 30 13:43:38.299801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.299436 5049 scope.go:122] "RemoveContainer" containerID="1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d" Oct 30 13:43:38.312384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.312359 5049 generic.go:356] "Generic (PLEG): container finished" podID="2f5bdbc6-d487-431f-a741-b47a041b1ca2" containerID="f4dffd41d091dfa01e3846e63eb850b90104396fb574e7564d8d1d0a29b38231" exitCode=0 Oct 30 13:43:38.312860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.312833 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" event={"ID":"2f5bdbc6-d487-431f-a741-b47a041b1ca2","Type":"ContainerDied","Data":"f4dffd41d091dfa01e3846e63eb850b90104396fb574e7564d8d1d0a29b38231"} Oct 30 13:43:38.337973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.337844 5049 scope.go:122] "RemoveContainer" containerID="1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d" Oct 30 13:43:38.338436 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:38.338406 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d\": container with ID starting with 1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d not found: ID does not exist" containerID="1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d" Oct 30 13:43:38.338584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.338436 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d"} err="failed to get container status \"1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d\": rpc error: code = NotFound desc = could not find container \"1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d\": container with ID starting with 1287524d24571be0a3f2e97f06228507f6953d4bdb152c6d3f03c2e9a151b15d not found: ID does not exist" Oct 30 13:43:38.339116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.339087 5049 generic.go:356] "Generic (PLEG): container finished" podID="2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd" containerID="ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034" exitCode=0 Oct 30 13:43:38.339220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.339142 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" Oct 30 13:43:38.339220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.339167 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" event={"ID":"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd","Type":"ContainerDied","Data":"ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034"} Oct 30 13:43:38.339311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.339227 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq" event={"ID":"2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd","Type":"ContainerDied","Data":"18d2ad9e13e2527e891af2a390286a6f021efe815d4da3ff9a07451e8327ba7f"} Oct 30 13:43:38.339311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.339258 5049 scope.go:122] "RemoveContainer" containerID="ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034" Oct 30 13:43:38.348565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.348528 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvl2n\" (UniqueName: \"kubernetes.io/projected/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd-kube-api-access-cvl2n\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:38.350988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.350963 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:38.351613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.351597 5049 scope.go:122] "RemoveContainer" containerID="ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034" Oct 30 13:43:38.351900 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:38.351880 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034\": container with ID starting with ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034 not found: ID does not exist" containerID="ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034" Oct 30 13:43:38.351978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.351907 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034"} err="failed to get container status \"ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034\": rpc error: code = NotFound desc = could not find container \"ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034\": container with ID starting with ad33973043cfbe814124dfc976f4583dd03dd617809b895fc6dddb0867756034 not found: ID does not exist" Oct 30 13:43:38.369766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.369743 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls"] Oct 30 13:43:38.399288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.399223 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ee35e-5859d7f5db-8fvls"] Oct 30 13:43:38.473120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.473075 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc"] Oct 30 13:43:38.476752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.476723 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ee35e-7jhwc"] Oct 30 13:43:38.483731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.483703 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq"] Oct 30 13:43:38.494130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.494065 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ee35e-6987bf95d7-q5skq"] Oct 30 13:43:38.546084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.546019 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:38.752916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:38.752813 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:39.109562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.109519 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:39.278509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.278471 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp"] Oct 30 13:43:39.337570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.337539 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd" path="/var/lib/kubelet/pods/2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd/volumes" Oct 30 13:43:39.337997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.337983 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ffe634-75ee-4ed1-abea-7dda3aad4a0a" path="/var/lib/kubelet/pods/56ffe634-75ee-4ed1-abea-7dda3aad4a0a/volumes" Oct 30 13:43:39.338436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.338421 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bf63e0-9560-4894-84b6-1d5c399783a8" path="/var/lib/kubelet/pods/a5bf63e0-9560-4894-84b6-1d5c399783a8/volumes" Oct 30 13:43:39.496697 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.496658 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:39.570309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.570271 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:39.673059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.673022 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:39.678337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.678296 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5bdbc6-d487-431f-a741-b47a041b1ca2-kube-api-access-kjtqm\" (UniqueName: \"kubernetes.io/projected/2f5bdbc6-d487-431f-a741-b47a041b1ca2-kube-api-access-kjtqm\") pod \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " Oct 30 13:43:39.678498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.678471 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-openshift-ca\") pod \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " Oct 30 13:43:39.679500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.678531 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-xks-ca\") pod \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\" (UID: \"2f5bdbc6-d487-431f-a741-b47a041b1ca2\") " Oct 30 13:43:39.679500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.678930 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-openshift-ca" pod "2f5bdbc6-d487-431f-a741-b47a041b1ca2" (UID: "2f5bdbc6-d487-431f-a741-b47a041b1ca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:39.679500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.679173 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:39.681168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.681133 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5bdbc6-d487-431f-a741-b47a041b1ca2-kube-api-access-kjtqm" pod "2f5bdbc6-d487-431f-a741-b47a041b1ca2" (UID: "2f5bdbc6-d487-431f-a741-b47a041b1ca2"). InnerVolumeSpecName "kube-api-access-kjtqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:39.681473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.681447 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-xks-ca" pod "2f5bdbc6-d487-431f-a741-b47a041b1ca2" (UID: "2f5bdbc6-d487-431f-a741-b47a041b1ca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:39.780982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.780949 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjtqm\" (UniqueName: \"kubernetes.io/projected/2f5bdbc6-d487-431f-a741-b47a041b1ca2-kube-api-access-kjtqm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:39.780982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.780977 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5bdbc6-d487-431f-a741-b47a041b1ca2-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:39.855143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:39.855105 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:40.360088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.360056 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" Oct 30 13:43:40.360088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.360080 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9" event={"ID":"2f5bdbc6-d487-431f-a741-b47a041b1ca2","Type":"ContainerDied","Data":"d20877a4885e5542e9f074be45e309b0f696dc45b5ee4d2a12ec9d3f4b2163d2"} Oct 30 13:43:40.360525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.360128 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d20877a4885e5542e9f074be45e309b0f696dc45b5ee4d2a12ec9d3f4b2163d2" Oct 30 13:43:40.574126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.574064 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:43:40.591211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591183 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a56f54-da40-419f-947b-34164430f3cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a56f54-da40-419f-947b-34164430f3cc-istiod-ca-cert\") pod \"22a56f54-da40-419f-947b-34164430f3cc\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " Oct 30 13:43:40.591401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591256 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-socket\") pod \"22a56f54-da40-419f-947b-34164430f3cc\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " Oct 30 13:43:40.591401 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591359 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-certs\") pod \"22a56f54-da40-419f-947b-34164430f3cc\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " Oct 30 13:43:40.591552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591422 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-data\") pod \"22a56f54-da40-419f-947b-34164430f3cc\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " Oct 30 13:43:40.591552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591464 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22a56f54-da40-419f-947b-34164430f3cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a56f54-da40-419f-947b-34164430f3cc-istio-podinfo\") pod \"22a56f54-da40-419f-947b-34164430f3cc\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " Oct 30 13:43:40.591552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591530 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-kube-api-access-kjpqs\" (UniqueName: \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-kube-api-access-kjpqs\") pod \"22a56f54-da40-419f-947b-34164430f3cc\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " Oct 30 13:43:40.591728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591584 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-socket" pod "22a56f54-da40-419f-947b-34164430f3cc" (UID: "22a56f54-da40-419f-947b-34164430f3cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:40.591728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591653 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-credential-socket\") pod \"22a56f54-da40-419f-947b-34164430f3cc\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " Oct 30 13:43:40.591728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591654 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a56f54-da40-419f-947b-34164430f3cc-istiod-ca-cert" pod "22a56f54-da40-419f-947b-34164430f3cc" (UID: "22a56f54-da40-419f-947b-34164430f3cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:40.591863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591747 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-istio-token\" (UniqueName: \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-istio-token\") pod \"22a56f54-da40-419f-947b-34164430f3cc\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " Oct 30 13:43:40.591863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591691 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-certs" pod "22a56f54-da40-419f-947b-34164430f3cc" (UID: "22a56f54-da40-419f-947b-34164430f3cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:40.591972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591861 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-envoy\") pod \"22a56f54-da40-419f-947b-34164430f3cc\" (UID: \"22a56f54-da40-419f-947b-34164430f3cc\") " Oct 30 13:43:40.592035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591743 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-data" pod "22a56f54-da40-419f-947b-34164430f3cc" (UID: "22a56f54-da40-419f-947b-34164430f3cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:40.592097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.591936 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-credential-socket" pod "22a56f54-da40-419f-947b-34164430f3cc" (UID: "22a56f54-da40-419f-947b-34164430f3cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:40.592521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.592484 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:40.592521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.592509 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:40.592521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.592518 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a56f54-da40-419f-947b-34164430f3cc-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:40.592521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.592527 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:40.592820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.592535 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:40.594061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.594032 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-envoy" pod "22a56f54-da40-419f-947b-34164430f3cc" (UID: "22a56f54-da40-419f-947b-34164430f3cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:40.594385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.594362 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22a56f54-da40-419f-947b-34164430f3cc-istio-podinfo" pod "22a56f54-da40-419f-947b-34164430f3cc" (UID: "22a56f54-da40-419f-947b-34164430f3cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:43:40.594464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.594413 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-istio-token" pod "22a56f54-da40-419f-947b-34164430f3cc" (UID: "22a56f54-da40-419f-947b-34164430f3cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:40.594464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.594435 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-kube-api-access-kjpqs" pod "22a56f54-da40-419f-947b-34164430f3cc" (UID: "22a56f54-da40-419f-947b-34164430f3cc"). InnerVolumeSpecName "kube-api-access-kjpqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:40.694495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.694453 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a56f54-da40-419f-947b-34164430f3cc-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:40.694495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.694491 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a56f54-da40-419f-947b-34164430f3cc-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:40.694762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.694506 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjpqs\" (UniqueName: \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-kube-api-access-kjpqs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:40.694762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.694522 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a56f54-da40-419f-947b-34164430f3cc-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:40.816525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:40.816491 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:41.382241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.382209 5049 generic.go:356] "Generic (PLEG): container finished" podID="22a56f54-da40-419f-947b-34164430f3cc" containerID="a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1" exitCode=0 Oct 30 13:43:41.382950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.382528 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" Oct 30 13:43:41.385203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.382549 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" event={"ID":"22a56f54-da40-419f-947b-34164430f3cc","Type":"ContainerDied","Data":"a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1"} Oct 30 13:43:41.385355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.385259 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl" event={"ID":"22a56f54-da40-419f-947b-34164430f3cc","Type":"ContainerDied","Data":"9c4441acf2f8ed9d2b2fa502c92891ee25526161b2b09017f1ab78b2a987b383"} Oct 30 13:43:41.385355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.385299 5049 scope.go:122] "RemoveContainer" containerID="a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1" Oct 30 13:43:41.402828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.402793 5049 scope.go:122] "RemoveContainer" containerID="51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e" Oct 30 13:43:41.417710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.417169 5049 scope.go:122] "RemoveContainer" containerID="a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1" Oct 30 13:43:41.417818 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:41.417751 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1\": container with ID starting with a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1 not found: ID does not exist" containerID="a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1" Oct 30 13:43:41.417818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.417787 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1"} err="failed to get container status \"a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1\": rpc error: code = NotFound desc = could not find container \"a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1\": container with ID starting with a50e43f32c70d7b6b6b9b23fb20f401cfa95d9486e79d830cb3cdd4d001679f1 not found: ID does not exist" Oct 30 13:43:41.417818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.417813 5049 scope.go:122] "RemoveContainer" containerID="51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e" Oct 30 13:43:41.418244 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:41.418146 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e\": container with ID starting with 51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e not found: ID does not exist" containerID="51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e" Oct 30 13:43:41.418244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.418188 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e"} err="failed to get container status \"51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e\": rpc error: code = NotFound desc = could not find container \"51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e\": container with ID starting with 51c8f89104be27b68b8cc70aec16faff35522feeb7fd3541f8abb669ee2e900e not found: ID does not exist" Oct 30 13:43:41.455721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.455670 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl"] Oct 30 13:43:41.488348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:41.485647 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e3ebf5-openshift-default-5d7944cf85-qlldl"] Oct 30 13:43:42.475465 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:42.475422 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:42.708503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:42.708470 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:42.896765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:42.896729 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:43.077719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.077682 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:43.283362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.283250 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5"] Oct 30 13:43:43.284032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284015 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd" containerName="payload-pre-processing" Oct 30 13:43:43.284081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284033 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd" containerName="payload-pre-processing" Oct 30 13:43:43.284081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284059 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2f5bdbc6-d487-431f-a741-b47a041b1ca2" containerName="revoke-keys" Oct 30 13:43:43.284081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284064 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5bdbc6-d487-431f-a741-b47a041b1ca2" containerName="revoke-keys" Oct 30 13:43:43.284081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284072 5049 cpu_manager.go:414] "containerMap: removing container" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" Oct 30 13:43:43.284081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284077 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" Oct 30 13:43:43.284233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284092 5049 cpu_manager.go:414] "containerMap: removing container" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" Oct 30 13:43:43.284233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284098 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" Oct 30 13:43:43.284233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284167 5049 cpu_manager.go:414] "containerMap: removing container" podUID="56ffe634-75ee-4ed1-abea-7dda3aad4a0a" containerName="payload-processing" Oct 30 13:43:43.284233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284173 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ffe634-75ee-4ed1-abea-7dda3aad4a0a" containerName="payload-processing" Oct 30 13:43:43.284378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284267 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ffe634-75ee-4ed1-abea-7dda3aad4a0a" containerName="payload-processing" Oct 30 13:43:43.284378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284276 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5e4b9d-f3a9-40d8-9eac-30218c27dcfd" containerName="payload-pre-processing" Oct 30 13:43:43.284378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284285 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5bdbc6-d487-431f-a741-b47a041b1ca2" containerName="revoke-keys" Oct 30 13:43:43.284378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284292 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" Oct 30 13:43:43.284378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.284299 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a56f54-da40-419f-947b-34164430f3cc" containerName="istio-proxy" Oct 30 13:43:43.289314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.289294 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.309217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.309185 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5"] Oct 30 13:43:43.314993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.314967 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:43:43.315276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.315258 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:43:43.337002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.336970 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.337232 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.337215 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtrd4\" (UniqueName: \"kubernetes.io/projected/2aaf5eb0-8e93-4671-93ab-34d20e197783-kube-api-access-vtrd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.337393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.337290 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.338849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.338821 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a56f54-da40-419f-947b-34164430f3cc" path="/var/lib/kubelet/pods/22a56f54-da40-419f-947b-34164430f3cc/volumes" Oct 30 13:43:43.438662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.438619 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtrd4\" (UniqueName: \"kubernetes.io/projected/2aaf5eb0-8e93-4671-93ab-34d20e197783-kube-api-access-vtrd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.438871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.438679 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.438871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.438761 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.439552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.439525 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.441249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.441224 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.461949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.461915 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtrd4\" (UniqueName: \"kubernetes.io/projected/2aaf5eb0-8e93-4671-93ab-34d20e197783-kube-api-access-vtrd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.603296 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.603267 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:43.920807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:43.920771 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5"] Oct 30 13:43:43.923050 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:43:43.923023 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aaf5eb0_8e93_4671_93ab_34d20e197783.slice/crio-4af47864e3a2456a462cf9f36ef2ce0a9512a599f8c92ca03d76e858cd1cebdf WatchSource:0}: Error finding container 4af47864e3a2456a462cf9f36ef2ce0a9512a599f8c92ca03d76e858cd1cebdf: Status 404 returned error can't find the container with id 4af47864e3a2456a462cf9f36ef2ce0a9512a599f8c92ca03d76e858cd1cebdf Oct 30 13:43:44.279502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:44.279452 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" podUID="e0fd7d58-f16f-47d9-8265-0cb67014efa1" containerName="payload-processing" containerID="cri-o://3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d" gracePeriod=25 Oct 30 13:43:44.409582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:44.409533 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" event={"ID":"2aaf5eb0-8e93-4671-93ab-34d20e197783","Type":"ContainerStarted","Data":"a7b9994a25b9c274356dc91c8cbb925ff877b1c595a51fe5b859ca40a137781f"} Oct 30 13:43:44.409794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:44.409603 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" event={"ID":"2aaf5eb0-8e93-4671-93ab-34d20e197783","Type":"ContainerStarted","Data":"4af47864e3a2456a462cf9f36ef2ce0a9512a599f8c92ca03d76e858cd1cebdf"} Oct 30 13:43:44.458624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:44.458570 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" podStartSLOduration=1.458555866 podStartE2EDuration="1.458555866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:44.454476118 +0000 UTC m=+3131.269670526" watchObservedRunningTime="2026-10-30 13:43:44.458555866 +0000 UTC m=+3131.273750288" Oct 30 13:43:45.423819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:45.423783 5049 generic.go:356] "Generic (PLEG): container finished" podID="2aaf5eb0-8e93-4671-93ab-34d20e197783" containerID="a7b9994a25b9c274356dc91c8cbb925ff877b1c595a51fe5b859ca40a137781f" exitCode=0 Oct 30 13:43:45.424253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:45.423877 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" event={"ID":"2aaf5eb0-8e93-4671-93ab-34d20e197783","Type":"ContainerDied","Data":"a7b9994a25b9c274356dc91c8cbb925ff877b1c595a51fe5b859ca40a137781f"} Oct 30 13:43:45.825910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:45.825865 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:46.205178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.205014 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:46.246519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.246485 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h"] Oct 30 13:43:46.267827 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.267797 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb"] Oct 30 13:43:46.377459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.377427 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:46.503446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.497435 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds"] Oct 30 13:43:46.503446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.497710 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" containerID="cri-o://bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3" gracePeriod=30 Oct 30 13:43:46.504047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.503559 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88"] Oct 30 13:43:46.504047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.503905 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" containerID="cri-o://229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01" gracePeriod=30 Oct 30 13:43:46.604204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.603969 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:46.749723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.749693 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:46.780554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.780468 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:46.808777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.808739 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aaf5eb0-8e93-4671-93ab-34d20e197783-kube-api-access-vtrd4\" (UniqueName: \"kubernetes.io/projected/2aaf5eb0-8e93-4671-93ab-34d20e197783-kube-api-access-vtrd4\") pod \"2aaf5eb0-8e93-4671-93ab-34d20e197783\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " Oct 30 13:43:46.808964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.808909 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-xks-ca\") pod \"2aaf5eb0-8e93-4671-93ab-34d20e197783\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " Oct 30 13:43:46.809088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.809071 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-openshift-ca\") pod \"2aaf5eb0-8e93-4671-93ab-34d20e197783\" (UID: \"2aaf5eb0-8e93-4671-93ab-34d20e197783\") " Oct 30 13:43:46.812618 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.812586 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-openshift-ca" pod "2aaf5eb0-8e93-4671-93ab-34d20e197783" (UID: "2aaf5eb0-8e93-4671-93ab-34d20e197783"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:46.819722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.819696 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-xks-ca" pod "2aaf5eb0-8e93-4671-93ab-34d20e197783" (UID: "2aaf5eb0-8e93-4671-93ab-34d20e197783"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:46.821255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.821231 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aaf5eb0-8e93-4671-93ab-34d20e197783-kube-api-access-vtrd4" pod "2aaf5eb0-8e93-4671-93ab-34d20e197783" (UID: "2aaf5eb0-8e93-4671-93ab-34d20e197783"). InnerVolumeSpecName "kube-api-access-vtrd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:46.917653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.917608 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:46.917653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.917644 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaf5eb0-8e93-4671-93ab-34d20e197783-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:46.917949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.917666 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtrd4\" (UniqueName: \"kubernetes.io/projected/2aaf5eb0-8e93-4671-93ab-34d20e197783-kube-api-access-vtrd4\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:46.996396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.996359 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh"] Oct 30 13:43:46.997184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.997165 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2aaf5eb0-8e93-4671-93ab-34d20e197783" containerName="revoke-keys" Oct 30 13:43:46.997184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.997184 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aaf5eb0-8e93-4671-93ab-34d20e197783" containerName="revoke-keys" Oct 30 13:43:46.997402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:46.997387 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aaf5eb0-8e93-4671-93ab-34d20e197783" containerName="revoke-keys" Oct 30 13:43:47.001080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.001060 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.022375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.022004 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh"] Oct 30 13:43:47.121241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.121193 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.121445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.121404 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.121579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.121560 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mdkx\" (UniqueName: \"kubernetes.io/projected/77a71238-c862-4567-96ab-57bec5c3840e-kube-api-access-6mdkx\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.223368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.223311 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mdkx\" (UniqueName: \"kubernetes.io/projected/77a71238-c862-4567-96ab-57bec5c3840e-kube-api-access-6mdkx\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.223575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.223406 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.223575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.223527 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.224229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.224195 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.225966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.225939 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.255519 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.255480 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mdkx\" (UniqueName: \"kubernetes.io/projected/77a71238-c862-4567-96ab-57bec5c3840e-kube-api-access-6mdkx\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.294615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.294577 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:47.318345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.318294 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:47.447828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.447783 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" Oct 30 13:43:47.447828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.447802 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5" event={"ID":"2aaf5eb0-8e93-4671-93ab-34d20e197783","Type":"ContainerDied","Data":"4af47864e3a2456a462cf9f36ef2ce0a9512a599f8c92ca03d76e858cd1cebdf"} Oct 30 13:43:47.448076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.447867 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4af47864e3a2456a462cf9f36ef2ce0a9512a599f8c92ca03d76e858cd1cebdf" Oct 30 13:43:47.660501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.660421 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b"] Oct 30 13:43:47.660940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.660615 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" podUID="805a405c-36db-43a0-b270-d33398a5d952" containerName="istio-proxy" containerID="cri-o://4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8" gracePeriod=30 Oct 30 13:43:47.675935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.675910 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh"] Oct 30 13:43:47.676547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.676520 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:47.678975 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:43:47.678953 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a71238_c862_4567_96ab_57bec5c3840e.slice/crio-d857c3f8d81d0795ba20dcf82c3eb4bbd537b02dc2007590309d1ff7daa81b8c WatchSource:0}: Error finding container d857c3f8d81d0795ba20dcf82c3eb4bbd537b02dc2007590309d1ff7daa81b8c: Status 404 returned error can't find the container with id d857c3f8d81d0795ba20dcf82c3eb4bbd537b02dc2007590309d1ff7daa81b8c Oct 30 13:43:47.846210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.846173 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:47.907566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:47.907524 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" podUID="e0fd7d58-f16f-47d9-8265-0cb67014efa1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:43:48.004125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.004092 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:48.121840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.121811 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb"] Oct 30 13:43:48.126077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.126055 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.153158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.152433 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb"] Oct 30 13:43:48.237667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.237570 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.237667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.237660 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l47fc\" (UniqueName: \"kubernetes.io/projected/3c1e5505-dd44-45c7-9745-ebf8052504a0-kube-api-access-l47fc\") pod \"maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.237929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.237754 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.249757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.249722 5049 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:43:48.249940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.249782 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" podUID="805a405c-36db-43a0-b270-d33398a5d952" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:43:48.339275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.339240 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.339496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.339431 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.339496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.339471 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l47fc\" (UniqueName: \"kubernetes.io/projected/3c1e5505-dd44-45c7-9745-ebf8052504a0-kube-api-access-l47fc\") pod \"maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.340002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.339979 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.341928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.341898 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.361416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.361387 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l47fc\" (UniqueName: \"kubernetes.io/projected/3c1e5505-dd44-45c7-9745-ebf8052504a0-kube-api-access-l47fc\") pod \"maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.440735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.440701 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:48.458154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.458119 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" event={"ID":"77a71238-c862-4567-96ab-57bec5c3840e","Type":"ContainerStarted","Data":"4a1e90f7bd396ba5f2e98e2bc819e03d185f76a0b36b66b3282936ae6efa6599"} Oct 30 13:43:48.458289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.458177 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" event={"ID":"77a71238-c862-4567-96ab-57bec5c3840e","Type":"ContainerStarted","Data":"d857c3f8d81d0795ba20dcf82c3eb4bbd537b02dc2007590309d1ff7daa81b8c"} Oct 30 13:43:48.497735 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.497635 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" podStartSLOduration=2.49762166 podStartE2EDuration="2.49762166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:48.494216719 +0000 UTC m=+3135.309411141" watchObservedRunningTime="2026-10-30 13:43:48.49762166 +0000 UTC m=+3135.312816082" Oct 30 13:43:48.761752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:48.761728 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb"] Oct 30 13:43:48.824404 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:43:48.824353 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1e5505_dd44_45c7_9745_ebf8052504a0.slice/crio-bf0eea3be48461ec842af16843837bdad2ebdd6372bb7f6a1d29138f440077b5 WatchSource:0}: Error finding container bf0eea3be48461ec842af16843837bdad2ebdd6372bb7f6a1d29138f440077b5: Status 404 returned error can't find the container with id bf0eea3be48461ec842af16843837bdad2ebdd6372bb7f6a1d29138f440077b5 Oct 30 13:43:49.468240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:49.468203 5049 generic.go:356] "Generic (PLEG): container finished" podID="77a71238-c862-4567-96ab-57bec5c3840e" containerID="4a1e90f7bd396ba5f2e98e2bc819e03d185f76a0b36b66b3282936ae6efa6599" exitCode=0 Oct 30 13:43:49.468427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:49.468284 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" event={"ID":"77a71238-c862-4567-96ab-57bec5c3840e","Type":"ContainerDied","Data":"4a1e90f7bd396ba5f2e98e2bc819e03d185f76a0b36b66b3282936ae6efa6599"} Oct 30 13:43:49.470105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:49.470079 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" event={"ID":"3c1e5505-dd44-45c7-9745-ebf8052504a0","Type":"ContainerStarted","Data":"240bc13a69eff835d983881e8ff6804712ef7ef96e08e06acc0be3b310a72c0f"} Oct 30 13:43:49.470221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:49.470121 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" event={"ID":"3c1e5505-dd44-45c7-9745-ebf8052504a0","Type":"ContainerStarted","Data":"bf0eea3be48461ec842af16843837bdad2ebdd6372bb7f6a1d29138f440077b5"} Oct 30 13:43:49.551753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:49.551686 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" podStartSLOduration=1.551666387 podStartE2EDuration="1.551666387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:49.545463108 +0000 UTC m=+3136.360657556" watchObservedRunningTime="2026-10-30 13:43:49.551666387 +0000 UTC m=+3136.366860812" Oct 30 13:43:49.935411 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:49.935367 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:50.097120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.097086 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:43:50.097298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.097139 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:43:50.481425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.481303 5049 generic.go:356] "Generic (PLEG): container finished" podID="3c1e5505-dd44-45c7-9745-ebf8052504a0" containerID="240bc13a69eff835d983881e8ff6804712ef7ef96e08e06acc0be3b310a72c0f" exitCode=0 Oct 30 13:43:50.481425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.481372 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" event={"ID":"3c1e5505-dd44-45c7-9745-ebf8052504a0","Type":"ContainerDied","Data":"240bc13a69eff835d983881e8ff6804712ef7ef96e08e06acc0be3b310a72c0f"} Oct 30 13:43:50.671280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.671251 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:50.773489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.773409 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a71238-c862-4567-96ab-57bec5c3840e-kube-api-access-6mdkx\" (UniqueName: \"kubernetes.io/projected/77a71238-c862-4567-96ab-57bec5c3840e-kube-api-access-6mdkx\") pod \"77a71238-c862-4567-96ab-57bec5c3840e\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " Oct 30 13:43:50.773489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.773474 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-openshift-ca\") pod \"77a71238-c862-4567-96ab-57bec5c3840e\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " Oct 30 13:43:50.773737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.773515 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-xks-ca\") pod \"77a71238-c862-4567-96ab-57bec5c3840e\" (UID: \"77a71238-c862-4567-96ab-57bec5c3840e\") " Oct 30 13:43:50.773902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.773876 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-openshift-ca" pod "77a71238-c862-4567-96ab-57bec5c3840e" (UID: "77a71238-c862-4567-96ab-57bec5c3840e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:50.774217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.774198 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:50.775692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.775664 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-xks-ca" pod "77a71238-c862-4567-96ab-57bec5c3840e" (UID: "77a71238-c862-4567-96ab-57bec5c3840e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:50.775821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.775746 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a71238-c862-4567-96ab-57bec5c3840e-kube-api-access-6mdkx" pod "77a71238-c862-4567-96ab-57bec5c3840e" (UID: "77a71238-c862-4567-96ab-57bec5c3840e"). InnerVolumeSpecName "kube-api-access-6mdkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:50.875767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.875733 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mdkx\" (UniqueName: \"kubernetes.io/projected/77a71238-c862-4567-96ab-57bec5c3840e-kube-api-access-6mdkx\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:50.875767 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:50.875760 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a71238-c862-4567-96ab-57bec5c3840e-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.247126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.247087 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" podUID="6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e" containerName="payload-processing" containerID="cri-o://874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08" gracePeriod=25 Oct 30 13:43:51.268846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.268812 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" podUID="22699b7b-4d79-427b-a469-a8d2d87a7e51" containerName="payload-pre-processing" containerID="cri-o://66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c" gracePeriod=25 Oct 30 13:43:51.491958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.491930 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" Oct 30 13:43:51.492124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.491963 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh" event={"ID":"77a71238-c862-4567-96ab-57bec5c3840e","Type":"ContainerDied","Data":"d857c3f8d81d0795ba20dcf82c3eb4bbd537b02dc2007590309d1ff7daa81b8c"} Oct 30 13:43:51.492124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.492012 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d857c3f8d81d0795ba20dcf82c3eb4bbd537b02dc2007590309d1ff7daa81b8c" Oct 30 13:43:51.678821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.678790 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:51.792739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.790203 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1e5505-dd44-45c7-9745-ebf8052504a0-kube-api-access-l47fc\" (UniqueName: \"kubernetes.io/projected/3c1e5505-dd44-45c7-9745-ebf8052504a0-kube-api-access-l47fc\") pod \"3c1e5505-dd44-45c7-9745-ebf8052504a0\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " Oct 30 13:43:51.792739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.790246 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-openshift-ca\") pod \"3c1e5505-dd44-45c7-9745-ebf8052504a0\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " Oct 30 13:43:51.792739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.790468 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-xks-ca\") pod \"3c1e5505-dd44-45c7-9745-ebf8052504a0\" (UID: \"3c1e5505-dd44-45c7-9745-ebf8052504a0\") " Oct 30 13:43:51.792739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.790898 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-openshift-ca" pod "3c1e5505-dd44-45c7-9745-ebf8052504a0" (UID: "3c1e5505-dd44-45c7-9745-ebf8052504a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:51.792739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.791105 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.794355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.793835 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1e5505-dd44-45c7-9745-ebf8052504a0-kube-api-access-l47fc" pod "3c1e5505-dd44-45c7-9745-ebf8052504a0" (UID: "3c1e5505-dd44-45c7-9745-ebf8052504a0"). InnerVolumeSpecName "kube-api-access-l47fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:51.795022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.794999 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-xks-ca" pod "3c1e5505-dd44-45c7-9745-ebf8052504a0" (UID: "3c1e5505-dd44-45c7-9745-ebf8052504a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:51.839161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.839128 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:43:51.845580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.845555 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:43:51.892906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.892871 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l47fc\" (UniqueName: \"kubernetes.io/projected/3c1e5505-dd44-45c7-9745-ebf8052504a0-kube-api-access-l47fc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.892906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.892907 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1e5505-dd44-45c7-9745-ebf8052504a0-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.993405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993370 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-credential-socket\") pod \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " Oct 30 13:43:51.993578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993414 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-kube-api-access-v4b7d\" (UniqueName: \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-kube-api-access-v4b7d\") pod \"46ab940e-48f1-4439-8842-e7bbba2b45b7\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " Oct 30 13:43:51.993578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993450 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-envoy\") pod \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " Oct 30 13:43:51.993578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993521 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-podinfo\") pod \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " Oct 30 13:43:51.993578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993563 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-data\") pod \"46ab940e-48f1-4439-8842-e7bbba2b45b7\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " Oct 30 13:43:51.993813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993595 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ab940e-48f1-4439-8842-e7bbba2b45b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ab940e-48f1-4439-8842-e7bbba2b45b7-istiod-ca-cert\") pod \"46ab940e-48f1-4439-8842-e7bbba2b45b7\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " Oct 30 13:43:51.993813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993670 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-credential-socket" pod "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" (UID: "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.993813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993688 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-kube-api-access-mzzrc\" (UniqueName: \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-kube-api-access-mzzrc\") pod \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " Oct 30 13:43:51.993813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993730 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istiod-ca-cert\") pod \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " Oct 30 13:43:51.993813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993763 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-socket\") pod \"46ab940e-48f1-4439-8842-e7bbba2b45b7\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " Oct 30 13:43:51.993813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993804 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-token\" (UniqueName: \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-token\") pod \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " Oct 30 13:43:51.994156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993837 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-certs\") pod \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " Oct 30 13:43:51.994156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993888 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-socket\") pod \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " Oct 30 13:43:51.994156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993936 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-credential-socket\") pod \"46ab940e-48f1-4439-8842-e7bbba2b45b7\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " Oct 30 13:43:51.994156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.993975 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-certs\") pod \"46ab940e-48f1-4439-8842-e7bbba2b45b7\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " Oct 30 13:43:51.994156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994043 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-token\" (UniqueName: \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-token\") pod \"46ab940e-48f1-4439-8842-e7bbba2b45b7\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " Oct 30 13:43:51.994156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994099 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-podinfo\") pod \"46ab940e-48f1-4439-8842-e7bbba2b45b7\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " Oct 30 13:43:51.994156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994125 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-data" pod "46ab940e-48f1-4439-8842-e7bbba2b45b7" (UID: "46ab940e-48f1-4439-8842-e7bbba2b45b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.994562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994166 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-data\") pod \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\" (UID: \"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4\") " Oct 30 13:43:51.994562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994225 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-envoy\") pod \"46ab940e-48f1-4439-8842-e7bbba2b45b7\" (UID: \"46ab940e-48f1-4439-8842-e7bbba2b45b7\") " Oct 30 13:43:51.994562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994252 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-certs" pod "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" (UID: "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.994562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994284 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ab940e-48f1-4439-8842-e7bbba2b45b7-istiod-ca-cert" pod "46ab940e-48f1-4439-8842-e7bbba2b45b7" (UID: "46ab940e-48f1-4439-8842-e7bbba2b45b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:51.994562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994540 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-certs" pod "46ab940e-48f1-4439-8842-e7bbba2b45b7" (UID: "46ab940e-48f1-4439-8842-e7bbba2b45b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.994855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994626 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istiod-ca-cert" pod "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" (UID: "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:51.994855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994719 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-socket" pod "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" (UID: "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.994977 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994905 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-credential-socket" pod "46ab940e-48f1-4439-8842-e7bbba2b45b7" (UID: "46ab940e-48f1-4439-8842-e7bbba2b45b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.995039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.994970 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-socket" pod "46ab940e-48f1-4439-8842-e7bbba2b45b7" (UID: "46ab940e-48f1-4439-8842-e7bbba2b45b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.995100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.995078 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-data" pod "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" (UID: "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.995730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.995706 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.995730 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.995732 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.996025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.995747 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.996025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.995762 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.996025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.995777 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.996025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.995792 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.996025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.995807 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.996025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.995821 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ab940e-48f1-4439-8842-e7bbba2b45b7-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.997056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.997036 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-token" pod "46ab940e-48f1-4439-8842-e7bbba2b45b7" (UID: "46ab940e-48f1-4439-8842-e7bbba2b45b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:51.997628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.997598 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-podinfo" pod "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" (UID: "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:43:51.997628 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.997603 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-envoy" pod "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" (UID: "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.997783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.997638 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-podinfo" pod "46ab940e-48f1-4439-8842-e7bbba2b45b7" (UID: "46ab940e-48f1-4439-8842-e7bbba2b45b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:43:51.997783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.997712 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-kube-api-access-v4b7d" pod "46ab940e-48f1-4439-8842-e7bbba2b45b7" (UID: "46ab940e-48f1-4439-8842-e7bbba2b45b7"). InnerVolumeSpecName "kube-api-access-v4b7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:51.998313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.998293 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-kube-api-access-mzzrc" pod "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" (UID: "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4"). InnerVolumeSpecName "kube-api-access-mzzrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:51.999004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.998974 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-envoy" pod "46ab940e-48f1-4439-8842-e7bbba2b45b7" (UID: "46ab940e-48f1-4439-8842-e7bbba2b45b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.999977 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:51.999959 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-token" pod "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" (UID: "69bb0776-30fd-4a9d-8d81-b1a16ee10fb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:52.036784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.036645 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:52.096682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.096647 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.096676 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.096684 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.096692 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.096701 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4b7d\" (UniqueName: \"kubernetes.io/projected/46ab940e-48f1-4439-8842-e7bbba2b45b7-kube-api-access-v4b7d\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.096710 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.096720 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.096728 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzzrc\" (UniqueName: \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-kube-api-access-mzzrc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.096736 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ab940e-48f1-4439-8842-e7bbba2b45b7-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.096745 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.502417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.502380 5049 generic.go:356] "Generic (PLEG): container finished" podID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerID="229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01" exitCode=0 Oct 30 13:43:52.502861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.502423 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" Oct 30 13:43:52.502861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.502442 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" event={"ID":"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4","Type":"ContainerDied","Data":"229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01"} Oct 30 13:43:52.502861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.502500 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88" event={"ID":"69bb0776-30fd-4a9d-8d81-b1a16ee10fb4","Type":"ContainerDied","Data":"dc6cd477d8674a2e64c71ae780ba7a96e74a4f6dc6092c9f25ac8a5655a20b76"} Oct 30 13:43:52.502861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.502533 5049 scope.go:122] "RemoveContainer" containerID="229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01" Oct 30 13:43:52.504529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.504502 5049 generic.go:356] "Generic (PLEG): container finished" podID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerID="bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3" exitCode=0 Oct 30 13:43:52.504687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.504554 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" Oct 30 13:43:52.504687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.504580 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" event={"ID":"46ab940e-48f1-4439-8842-e7bbba2b45b7","Type":"ContainerDied","Data":"bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3"} Oct 30 13:43:52.504687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.504632 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds" event={"ID":"46ab940e-48f1-4439-8842-e7bbba2b45b7","Type":"ContainerDied","Data":"97fbaf835ce8768d14ad19bcecea9f146897293da4e4e51d59a0b3dc70d54389"} Oct 30 13:43:52.511686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.506351 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" event={"ID":"3c1e5505-dd44-45c7-9745-ebf8052504a0","Type":"ContainerDied","Data":"bf0eea3be48461ec842af16843837bdad2ebdd6372bb7f6a1d29138f440077b5"} Oct 30 13:43:52.511686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.506384 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb" Oct 30 13:43:52.511686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.506396 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf0eea3be48461ec842af16843837bdad2ebdd6372bb7f6a1d29138f440077b5" Oct 30 13:43:52.521903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.521883 5049 scope.go:122] "RemoveContainer" containerID="229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01" Oct 30 13:43:52.522201 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:52.522182 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01\": container with ID starting with 229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01 not found: ID does not exist" containerID="229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01" Oct 30 13:43:52.522264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.522208 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01"} err="failed to get container status \"229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01\": rpc error: code = NotFound desc = could not find container \"229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01\": container with ID starting with 229d3ef522d657cb219b2db1f78415304f9a9a6af2c83e6e08f8d8c4e8b94e01 not found: ID does not exist" Oct 30 13:43:52.522264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.522224 5049 scope.go:122] "RemoveContainer" containerID="bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3" Oct 30 13:43:52.535945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.535570 5049 scope.go:122] "RemoveContainer" containerID="30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07" Oct 30 13:43:52.546516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.546494 5049 scope.go:122] "RemoveContainer" containerID="bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3" Oct 30 13:43:52.546840 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:52.546810 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3\": container with ID starting with bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3 not found: ID does not exist" containerID="bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3" Oct 30 13:43:52.546939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.546853 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3"} err="failed to get container status \"bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3\": rpc error: code = NotFound desc = could not find container \"bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3\": container with ID starting with bacf6d60731f71dda21a9a812575f3ceda9aee72efc405982f73976bee4b0bb3 not found: ID does not exist" Oct 30 13:43:52.546939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.546876 5049 scope.go:122] "RemoveContainer" containerID="30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07" Oct 30 13:43:52.547230 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:52.547203 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07\": container with ID starting with 30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07 not found: ID does not exist" containerID="30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07" Oct 30 13:43:52.547347 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.547238 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07"} err="failed to get container status \"30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07\": rpc error: code = NotFound desc = could not find container \"30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07\": container with ID starting with 30a40672d19169315de2117a8b9fc218a655979bed0889e872243aa194b62b07 not found: ID does not exist" Oct 30 13:43:52.547720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.547694 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" podUID="22699b7b-4d79-427b-a469-a8d2d87a7e51" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:43:52.547841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.547819 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" podUID="6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:43:52.556402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.556381 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88"] Oct 30 13:43:52.566308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.566285 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-zqz88"] Oct 30 13:43:52.617622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.617594 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds"] Oct 30 13:43:52.621868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.621842 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-vpfds"] Oct 30 13:43:52.948506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:52.948478 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:43:53.109637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.109602 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-credential-socket\") pod \"805a405c-36db-43a0-b270-d33398a5d952\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " Oct 30 13:43:53.109807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.109701 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-data\") pod \"805a405c-36db-43a0-b270-d33398a5d952\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " Oct 30 13:43:53.109807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.109739 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-kube-api-access-dp2sc\" (UniqueName: \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-kube-api-access-dp2sc\") pod \"805a405c-36db-43a0-b270-d33398a5d952\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " Oct 30 13:43:53.109807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.109786 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-envoy\") pod \"805a405c-36db-43a0-b270-d33398a5d952\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " Oct 30 13:43:53.110019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.109911 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805a405c-36db-43a0-b270-d33398a5d952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805a405c-36db-43a0-b270-d33398a5d952-istiod-ca-cert\") pod \"805a405c-36db-43a0-b270-d33398a5d952\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " Oct 30 13:43:53.110019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.109929 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-credential-socket" pod "805a405c-36db-43a0-b270-d33398a5d952" (UID: "805a405c-36db-43a0-b270-d33398a5d952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:53.110019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.109967 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-data" pod "805a405c-36db-43a0-b270-d33398a5d952" (UID: "805a405c-36db-43a0-b270-d33398a5d952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:53.110019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.109988 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-certs\") pod \"805a405c-36db-43a0-b270-d33398a5d952\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " Oct 30 13:43:53.110249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110029 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-socket\") pod \"805a405c-36db-43a0-b270-d33398a5d952\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " Oct 30 13:43:53.110249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110070 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/805a405c-36db-43a0-b270-d33398a5d952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805a405c-36db-43a0-b270-d33398a5d952-istio-podinfo\") pod \"805a405c-36db-43a0-b270-d33398a5d952\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " Oct 30 13:43:53.110249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110104 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-istio-token\" (UniqueName: \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-istio-token\") pod \"805a405c-36db-43a0-b270-d33398a5d952\" (UID: \"805a405c-36db-43a0-b270-d33398a5d952\") " Oct 30 13:43:53.110249 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110203 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-certs" pod "805a405c-36db-43a0-b270-d33398a5d952" (UID: "805a405c-36db-43a0-b270-d33398a5d952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:53.110499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110341 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-socket" pod "805a405c-36db-43a0-b270-d33398a5d952" (UID: "805a405c-36db-43a0-b270-d33398a5d952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:53.110499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110376 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805a405c-36db-43a0-b270-d33398a5d952-istiod-ca-cert" pod "805a405c-36db-43a0-b270-d33398a5d952" (UID: "805a405c-36db-43a0-b270-d33398a5d952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:53.110762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110741 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805a405c-36db-43a0-b270-d33398a5d952-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:53.110762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110765 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:53.110762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110781 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:53.110762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110794 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:53.111088 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.110808 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:53.112472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.112443 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-kube-api-access-dp2sc" pod "805a405c-36db-43a0-b270-d33398a5d952" (UID: "805a405c-36db-43a0-b270-d33398a5d952"). InnerVolumeSpecName "kube-api-access-dp2sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:53.112575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.112545 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-istio-token" pod "805a405c-36db-43a0-b270-d33398a5d952" (UID: "805a405c-36db-43a0-b270-d33398a5d952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:53.112646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.112628 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-envoy" pod "805a405c-36db-43a0-b270-d33398a5d952" (UID: "805a405c-36db-43a0-b270-d33398a5d952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:53.112880 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.112858 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/805a405c-36db-43a0-b270-d33398a5d952-istio-podinfo" pod "805a405c-36db-43a0-b270-d33398a5d952" (UID: "805a405c-36db-43a0-b270-d33398a5d952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:43:53.212202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.212169 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp2sc\" (UniqueName: \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-kube-api-access-dp2sc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:53.212202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.212197 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805a405c-36db-43a0-b270-d33398a5d952-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:53.212202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.212207 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805a405c-36db-43a0-b270-d33398a5d952-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:53.212492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.212215 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805a405c-36db-43a0-b270-d33398a5d952-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:53.346900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.337569 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" path="/var/lib/kubelet/pods/46ab940e-48f1-4439-8842-e7bbba2b45b7/volumes" Oct 30 13:43:53.346900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.338435 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" path="/var/lib/kubelet/pods/69bb0776-30fd-4a9d-8d81-b1a16ee10fb4/volumes" Oct 30 13:43:53.518268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.518182 5049 generic.go:356] "Generic (PLEG): container finished" podID="805a405c-36db-43a0-b270-d33398a5d952" containerID="4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8" exitCode=0 Oct 30 13:43:53.518268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.518241 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" event={"ID":"805a405c-36db-43a0-b270-d33398a5d952","Type":"ContainerDied","Data":"4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8"} Oct 30 13:43:53.518777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.518281 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" event={"ID":"805a405c-36db-43a0-b270-d33398a5d952","Type":"ContainerDied","Data":"f35fbde40272883428eb7369a9b3f93ceb5904aed6613178fb8eb505c0c5a165"} Oct 30 13:43:53.518777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.518302 5049 scope.go:122] "RemoveContainer" containerID="4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8" Oct 30 13:43:53.518777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.518243 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b" Oct 30 13:43:53.530422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.530402 5049 scope.go:122] "RemoveContainer" containerID="4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8" Oct 30 13:43:53.530705 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:53.530684 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8\": container with ID starting with 4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8 not found: ID does not exist" containerID="4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8" Oct 30 13:43:53.530778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.530713 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8"} err="failed to get container status \"4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8\": rpc error: code = NotFound desc = could not find container \"4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8\": container with ID starting with 4a1e62781bfc0ef2fc8cf77bc695b4cf6de988d12556c3e05350ec2da49574f8 not found: ID does not exist" Oct 30 13:43:53.566942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.566919 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b"] Oct 30 13:43:53.572706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.572685 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-cj28b"] Oct 30 13:43:53.677211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.677176 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559"] Oct 30 13:43:53.821824 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:53.821793 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:54.007003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:54.006970 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:55.338286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:55.338255 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805a405c-36db-43a0-b270-d33398a5d952" path="/var/lib/kubelet/pods/805a405c-36db-43a0-b270-d33398a5d952/volumes" Oct 30 13:43:55.708872 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:55.708768 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:56.677774 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:56.677736 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:43:57.567015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:57.566980 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:57.723648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:57.723575 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:57.906177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:57.906086 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" podUID="e0fd7d58-f16f-47d9-8265-0cb67014efa1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:43:58.678567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:58.678526 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" podUID="47d4c8d1-20dd-4057-9766-faca798308b8" containerName="payload-pre-processing" containerID="cri-o://fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857" gracePeriod=25 Oct 30 13:43:58.953172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:58.953146 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:43:58.991213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:58.991177 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d4c8d1-20dd-4057-9766-faca798308b8-kube-api-access-fzmzc\" (UniqueName: \"kubernetes.io/projected/47d4c8d1-20dd-4057-9766-faca798308b8-kube-api-access-fzmzc\") pod \"47d4c8d1-20dd-4057-9766-faca798308b8\" (UID: \"47d4c8d1-20dd-4057-9766-faca798308b8\") " Oct 30 13:43:58.991390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:58.991253 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d4c8d1-20dd-4057-9766-faca798308b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d4c8d1-20dd-4057-9766-faca798308b8-plugins-config-volume\") pod \"47d4c8d1-20dd-4057-9766-faca798308b8\" (UID: \"47d4c8d1-20dd-4057-9766-faca798308b8\") " Oct 30 13:43:58.991733 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:58.991702 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d4c8d1-20dd-4057-9766-faca798308b8-plugins-config-volume" pod "47d4c8d1-20dd-4057-9766-faca798308b8" (UID: "47d4c8d1-20dd-4057-9766-faca798308b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:58.991897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:58.991881 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d4c8d1-20dd-4057-9766-faca798308b8-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:58.994092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:58.994065 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d4c8d1-20dd-4057-9766-faca798308b8-kube-api-access-fzmzc" pod "47d4c8d1-20dd-4057-9766-faca798308b8" (UID: "47d4c8d1-20dd-4057-9766-faca798308b8"). InnerVolumeSpecName "kube-api-access-fzmzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:59.093130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.093088 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzmzc\" (UniqueName: \"kubernetes.io/projected/47d4c8d1-20dd-4057-9766-faca798308b8-kube-api-access-fzmzc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:43:59.399691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.399640 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:59.568008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.567976 5049 generic.go:356] "Generic (PLEG): container finished" podID="47d4c8d1-20dd-4057-9766-faca798308b8" containerID="fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857" exitCode=0 Oct 30 13:43:59.568212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.568019 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" Oct 30 13:43:59.568212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.568059 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" event={"ID":"47d4c8d1-20dd-4057-9766-faca798308b8","Type":"ContainerDied","Data":"fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857"} Oct 30 13:43:59.568212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.568116 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559" event={"ID":"47d4c8d1-20dd-4057-9766-faca798308b8","Type":"ContainerDied","Data":"297d608f06e8666b87d7016e65bcb63f131ae817bc38d9c9787983efec712777"} Oct 30 13:43:59.568212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.568145 5049 scope.go:122] "RemoveContainer" containerID="fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857" Oct 30 13:43:59.579956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.579928 5049 scope.go:122] "RemoveContainer" containerID="fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857" Oct 30 13:43:59.580223 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:43:59.580205 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857\": container with ID starting with fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857 not found: ID does not exist" containerID="fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857" Oct 30 13:43:59.580287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.580240 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857"} err="failed to get container status \"fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857\": rpc error: code = NotFound desc = could not find container \"fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857\": container with ID starting with fe805336e4c8648594181655acb9adde6194f696457c1e77b06403200fc8e857 not found: ID does not exist" Oct 30 13:43:59.622479 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.622396 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559"] Oct 30 13:43:59.622809 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:43:59.622786 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fdd4d4-7584bcfcc9-7t559"] Oct 30 13:44:01.052612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:01.052574 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:01.339387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:01.339291 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d4c8d1-20dd-4057-9766-faca798308b8" path="/var/lib/kubelet/pods/47d4c8d1-20dd-4057-9766-faca798308b8/volumes" Oct 30 13:44:01.684374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:01.684258 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:01.779558 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:01.779510 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j"] Oct 30 13:44:01.789462 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:01.789427 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b0ea9-g4j9j"] Oct 30 13:44:01.848286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:01.848247 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:02.320806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:02.320770 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:02.547310 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:02.547273 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" podUID="22699b7b-4d79-427b-a469-a8d2d87a7e51" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:02.547961 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:02.547934 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" podUID="6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:03.338371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:03.338316 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e0892b-70a8-4303-9e89-a62b1dce067d" path="/var/lib/kubelet/pods/89e0892b-70a8-4303-9e89-a62b1dce067d/volumes" Oct 30 13:44:03.810220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:03.810184 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:04.560578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.560550 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:44:04.611035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.611003 5049 generic.go:356] "Generic (PLEG): container finished" podID="e0fd7d58-f16f-47d9-8265-0cb67014efa1" containerID="3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d" exitCode=0 Oct 30 13:44:04.611212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.611066 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" Oct 30 13:44:04.611290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.611210 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" event={"ID":"e0fd7d58-f16f-47d9-8265-0cb67014efa1","Type":"ContainerDied","Data":"3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d"} Oct 30 13:44:04.611290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.611271 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp" event={"ID":"e0fd7d58-f16f-47d9-8265-0cb67014efa1","Type":"ContainerDied","Data":"6689aadf7e931bf5d14601dcb33f9d07515b30a4b618fd6ac71029f8490b7137"} Oct 30 13:44:04.611686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.611294 5049 scope.go:122] "RemoveContainer" containerID="3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d" Oct 30 13:44:04.657143 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.657116 5049 scope.go:122] "RemoveContainer" containerID="3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d" Oct 30 13:44:04.657644 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:04.657615 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d\": container with ID starting with 3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d not found: ID does not exist" containerID="3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d" Oct 30 13:44:04.657763 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.657652 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d"} err="failed to get container status \"3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d\": rpc error: code = NotFound desc = could not find container \"3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d\": container with ID starting with 3b1f1068d4bbd5856ef58cad5e4ce5a69c712faa7067955fc7c6423a4173da8d not found: ID does not exist" Oct 30 13:44:04.668268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.668234 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0fd7d58-f16f-47d9-8265-0cb67014efa1-kube-api-access-cvsms\" (UniqueName: \"kubernetes.io/projected/e0fd7d58-f16f-47d9-8265-0cb67014efa1-kube-api-access-cvsms\") pod \"e0fd7d58-f16f-47d9-8265-0cb67014efa1\" (UID: \"e0fd7d58-f16f-47d9-8265-0cb67014efa1\") " Oct 30 13:44:04.668449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.668356 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0fd7d58-f16f-47d9-8265-0cb67014efa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fd7d58-f16f-47d9-8265-0cb67014efa1-plugins-config-volume\") pod \"e0fd7d58-f16f-47d9-8265-0cb67014efa1\" (UID: \"e0fd7d58-f16f-47d9-8265-0cb67014efa1\") " Oct 30 13:44:04.668881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.668854 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fd7d58-f16f-47d9-8265-0cb67014efa1-plugins-config-volume" pod "e0fd7d58-f16f-47d9-8265-0cb67014efa1" (UID: "e0fd7d58-f16f-47d9-8265-0cb67014efa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:04.669162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.669141 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fd7d58-f16f-47d9-8265-0cb67014efa1-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:04.670648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.670627 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fd7d58-f16f-47d9-8265-0cb67014efa1-kube-api-access-cvsms" pod "e0fd7d58-f16f-47d9-8265-0cb67014efa1" (UID: "e0fd7d58-f16f-47d9-8265-0cb67014efa1"). InnerVolumeSpecName "kube-api-access-cvsms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:04.770081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.770044 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvsms\" (UniqueName: \"kubernetes.io/projected/e0fd7d58-f16f-47d9-8265-0cb67014efa1-kube-api-access-cvsms\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:04.956509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.956470 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp"] Oct 30 13:44:04.960680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:04.960649 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b0ea9-584685dfbb-rdgpp"] Oct 30 13:44:05.336558 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:05.336524 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0fd7d58-f16f-47d9-8265-0cb67014efa1" path="/var/lib/kubelet/pods/e0fd7d58-f16f-47d9-8265-0cb67014efa1/volumes" Oct 30 13:44:05.700494 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:05.700408 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:07.626226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:07.626195 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:09.075988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.075950 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb"] Oct 30 13:44:09.076456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.076252 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" podUID="134b56fe-3e2a-4794-9eff-feb1120babb1" containerName="maas-api" containerID="cri-o://37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9" gracePeriod=30 Oct 30 13:44:09.152679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.152639 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:09.360233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.360205 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:44:09.424089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.424053 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134b56fe-3e2a-4794-9eff-feb1120babb1-kube-api-access-vvcl6\" (UniqueName: \"kubernetes.io/projected/134b56fe-3e2a-4794-9eff-feb1120babb1-kube-api-access-vvcl6\") pod \"134b56fe-3e2a-4794-9eff-feb1120babb1\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " Oct 30 13:44:09.424286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.424147 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-maas-api-tls\") pod \"134b56fe-3e2a-4794-9eff-feb1120babb1\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " Oct 30 13:44:09.424286 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.424194 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134b56fe-3e2a-4794-9eff-feb1120babb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/134b56fe-3e2a-4794-9eff-feb1120babb1-openshift-service-ca\") pod \"134b56fe-3e2a-4794-9eff-feb1120babb1\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " Oct 30 13:44:09.424468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.424444 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-metrics-certs\") pod \"134b56fe-3e2a-4794-9eff-feb1120babb1\" (UID: \"134b56fe-3e2a-4794-9eff-feb1120babb1\") " Oct 30 13:44:09.424587 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.424534 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134b56fe-3e2a-4794-9eff-feb1120babb1-openshift-service-ca" pod "134b56fe-3e2a-4794-9eff-feb1120babb1" (UID: "134b56fe-3e2a-4794-9eff-feb1120babb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:09.425091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.425064 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/134b56fe-3e2a-4794-9eff-feb1120babb1-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:09.426550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.426523 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-maas-api-tls" pod "134b56fe-3e2a-4794-9eff-feb1120babb1" (UID: "134b56fe-3e2a-4794-9eff-feb1120babb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:09.429360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.426724 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134b56fe-3e2a-4794-9eff-feb1120babb1-kube-api-access-vvcl6" pod "134b56fe-3e2a-4794-9eff-feb1120babb1" (UID: "134b56fe-3e2a-4794-9eff-feb1120babb1"). InnerVolumeSpecName "kube-api-access-vvcl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:09.429360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.428567 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-metrics-certs" pod "134b56fe-3e2a-4794-9eff-feb1120babb1" (UID: "134b56fe-3e2a-4794-9eff-feb1120babb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:09.477614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.477533 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:09.526056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.526015 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:09.526056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.526056 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvcl6\" (UniqueName: \"kubernetes.io/projected/134b56fe-3e2a-4794-9eff-feb1120babb1-kube-api-access-vvcl6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:09.526304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.526072 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/134b56fe-3e2a-4794-9eff-feb1120babb1-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:09.546114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.546074 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w"] Oct 30 13:44:09.546301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.546271 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" podUID="8111c92e-8da9-49eb-ac5f-14427435e155" containerName="istio-proxy" containerID="cri-o://4d2c5f4703de51275fdb6aa5e549b87e61acd70e0ac016bf208674524b3ceb0a" gracePeriod=30 Oct 30 13:44:09.682226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.682126 5049 generic.go:356] "Generic (PLEG): container finished" podID="134b56fe-3e2a-4794-9eff-feb1120babb1" containerID="37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9" exitCode=0 Oct 30 13:44:09.682226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.682198 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" Oct 30 13:44:09.682502 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.682478 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" event={"ID":"134b56fe-3e2a-4794-9eff-feb1120babb1","Type":"ContainerDied","Data":"37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9"} Oct 30 13:44:09.682571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.682538 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb" event={"ID":"134b56fe-3e2a-4794-9eff-feb1120babb1","Type":"ContainerDied","Data":"5ba0985540dc94b90650a77e3b9734cf84970678e66fb470ba81d5971da826fa"} Oct 30 13:44:09.682633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.682569 5049 scope.go:122] "RemoveContainer" containerID="37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9" Oct 30 13:44:09.683064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.683032 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:09.715354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.710708 5049 scope.go:122] "RemoveContainer" containerID="37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9" Oct 30 13:44:09.715995 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:09.715908 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9\": container with ID starting with 37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9 not found: ID does not exist" containerID="37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9" Oct 30 13:44:09.715995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.715953 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9"} err="failed to get container status \"37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9\": rpc error: code = NotFound desc = could not find container \"37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9\": container with ID starting with 37dce2def45db971e225aa2076e1327da75fe94160b5569700b52ad8aec46ae9 not found: ID does not exist" Oct 30 13:44:09.749377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.742519 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb"] Oct 30 13:44:09.763399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.763367 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9cbcbc-86cbd4f988-6zdcb"] Oct 30 13:44:09.975062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:09.974970 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:10.092600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.092541 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9"] Oct 30 13:44:10.096357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.096309 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9cbcbc-lcmw9"] Oct 30 13:44:10.152895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.152861 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:10.219191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219154 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc"] Oct 30 13:44:10.219800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219784 5049 cpu_manager.go:414] "containerMap: removing container" podUID="805a405c-36db-43a0-b270-d33398a5d952" containerName="istio-proxy" Oct 30 13:44:10.219800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219800 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805a405c-36db-43a0-b270-d33398a5d952" containerName="istio-proxy" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219816 5049 cpu_manager.go:414] "containerMap: removing container" podUID="134b56fe-3e2a-4794-9eff-feb1120babb1" containerName="maas-api" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219821 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134b56fe-3e2a-4794-9eff-feb1120babb1" containerName="maas-api" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219837 5049 cpu_manager.go:414] "containerMap: removing container" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219844 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219859 5049 cpu_manager.go:414] "containerMap: removing container" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219864 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219877 5049 cpu_manager.go:414] "containerMap: removing container" podUID="77a71238-c862-4567-96ab-57bec5c3840e" containerName="revoke-keys" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219883 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a71238-c862-4567-96ab-57bec5c3840e" containerName="revoke-keys" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219896 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3c1e5505-dd44-45c7-9745-ebf8052504a0" containerName="revoke-keys" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219900 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1e5505-dd44-45c7-9745-ebf8052504a0" containerName="revoke-keys" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219907 5049 cpu_manager.go:414] "containerMap: removing container" podUID="47d4c8d1-20dd-4057-9766-faca798308b8" containerName="payload-pre-processing" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219911 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d4c8d1-20dd-4057-9766-faca798308b8" containerName="payload-pre-processing" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219924 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e0fd7d58-f16f-47d9-8265-0cb67014efa1" containerName="payload-processing" Oct 30 13:44:10.219954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.219929 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0fd7d58-f16f-47d9-8265-0cb67014efa1" containerName="payload-processing" Oct 30 13:44:10.220721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.220016 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" Oct 30 13:44:10.220721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.220026 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a71238-c862-4567-96ab-57bec5c3840e" containerName="revoke-keys" Oct 30 13:44:10.220721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.220033 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1e5505-dd44-45c7-9745-ebf8052504a0" containerName="revoke-keys" Oct 30 13:44:10.220721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.220039 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d4c8d1-20dd-4057-9766-faca798308b8" containerName="payload-pre-processing" Oct 30 13:44:10.220721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.220046 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0fd7d58-f16f-47d9-8265-0cb67014efa1" containerName="payload-processing" Oct 30 13:44:10.220721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.220051 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="805a405c-36db-43a0-b270-d33398a5d952" containerName="istio-proxy" Oct 30 13:44:10.220721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.220057 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="134b56fe-3e2a-4794-9eff-feb1120babb1" containerName="maas-api" Oct 30 13:44:10.220721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.220063 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="69bb0776-30fd-4a9d-8d81-b1a16ee10fb4" containerName="istio-proxy" Oct 30 13:44:10.224811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.224787 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.237790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.237771 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:44:10.240124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.240104 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:44:10.244442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.244357 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc"] Oct 30 13:44:10.339492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.339451 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.339665 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.339613 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbb8m\" (UniqueName: \"kubernetes.io/projected/2ac2db76-8e24-4899-aa7c-71755b76ead6-kube-api-access-pbb8m\") pod \"maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.339773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.339755 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.441253 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.441209 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.441476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.441316 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.441561 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.441474 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbb8m\" (UniqueName: \"kubernetes.io/projected/2ac2db76-8e24-4899-aa7c-71755b76ead6-kube-api-access-pbb8m\") pod \"maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.442218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.442193 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.443666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.443642 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.464176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.464146 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbb8m\" (UniqueName: \"kubernetes.io/projected/2ac2db76-8e24-4899-aa7c-71755b76ead6-kube-api-access-pbb8m\") pod \"maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.538881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.538793 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:10.675373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.675338 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:10.849593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:10.849571 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc"] Oct 30 13:44:10.852914 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:44:10.852886 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac2db76_8e24_4899_aa7c_71755b76ead6.slice/crio-ee59f77bf0599f847473e69d107f67263dbd872e0a84040865430696b7d2510f WatchSource:0}: Error finding container ee59f77bf0599f847473e69d107f67263dbd872e0a84040865430696b7d2510f: Status 404 returned error can't find the container with id ee59f77bf0599f847473e69d107f67263dbd872e0a84040865430696b7d2510f Oct 30 13:44:11.335570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.335538 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134b56fe-3e2a-4794-9eff-feb1120babb1" path="/var/lib/kubelet/pods/134b56fe-3e2a-4794-9eff-feb1120babb1/volumes" Oct 30 13:44:11.336026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.336011 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5bdbc6-d487-431f-a741-b47a041b1ca2" path="/var/lib/kubelet/pods/2f5bdbc6-d487-431f-a741-b47a041b1ca2/volumes" Oct 30 13:44:11.597831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.597804 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:44:11.601076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.601054 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:44:11.665679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.665596 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22699b7b-4d79-427b-a469-a8d2d87a7e51-kube-api-access-cbc92\" (UniqueName: \"kubernetes.io/projected/22699b7b-4d79-427b-a469-a8d2d87a7e51-kube-api-access-cbc92\") pod \"22699b7b-4d79-427b-a469-a8d2d87a7e51\" (UID: \"22699b7b-4d79-427b-a469-a8d2d87a7e51\") " Oct 30 13:44:11.665679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.665645 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-kube-api-access-lzcqg\" (UniqueName: \"kubernetes.io/projected/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-kube-api-access-lzcqg\") pod \"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e\" (UID: \"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e\") " Oct 30 13:44:11.665910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.665807 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22699b7b-4d79-427b-a469-a8d2d87a7e51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22699b7b-4d79-427b-a469-a8d2d87a7e51-plugins-config-volume\") pod \"22699b7b-4d79-427b-a469-a8d2d87a7e51\" (UID: \"22699b7b-4d79-427b-a469-a8d2d87a7e51\") " Oct 30 13:44:11.665971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.665930 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-plugins-config-volume\") pod \"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e\" (UID: \"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e\") " Oct 30 13:44:11.666273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.666241 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22699b7b-4d79-427b-a469-a8d2d87a7e51-plugins-config-volume" pod "22699b7b-4d79-427b-a469-a8d2d87a7e51" (UID: "22699b7b-4d79-427b-a469-a8d2d87a7e51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:11.666450 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.666281 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-plugins-config-volume" pod "6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e" (UID: "6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:11.666535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.666462 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:11.666535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.666476 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22699b7b-4d79-427b-a469-a8d2d87a7e51-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:11.667931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.667909 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22699b7b-4d79-427b-a469-a8d2d87a7e51-kube-api-access-cbc92" pod "22699b7b-4d79-427b-a469-a8d2d87a7e51" (UID: "22699b7b-4d79-427b-a469-a8d2d87a7e51"). InnerVolumeSpecName "kube-api-access-cbc92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:11.668039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.667941 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-kube-api-access-lzcqg" pod "6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e" (UID: "6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e"). InnerVolumeSpecName "kube-api-access-lzcqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:11.705588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.705549 5049 generic.go:356] "Generic (PLEG): container finished" podID="22699b7b-4d79-427b-a469-a8d2d87a7e51" containerID="66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c" exitCode=0 Oct 30 13:44:11.705746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.705652 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" Oct 30 13:44:11.705746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.705687 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" event={"ID":"22699b7b-4d79-427b-a469-a8d2d87a7e51","Type":"ContainerDied","Data":"66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c"} Oct 30 13:44:11.705746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.705739 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb" event={"ID":"22699b7b-4d79-427b-a469-a8d2d87a7e51","Type":"ContainerDied","Data":"ed39d7efa8e695881b3bfd49418609ef003a81f2a19a997d832df69d7fcc3923"} Oct 30 13:44:11.705930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.705771 5049 scope.go:122] "RemoveContainer" containerID="66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c" Oct 30 13:44:11.713883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.713847 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" event={"ID":"2ac2db76-8e24-4899-aa7c-71755b76ead6","Type":"ContainerStarted","Data":"9691f7708a2f46f6d45527fbcbfc4fd10111d487f13b3617bea37a5cae454b71"} Oct 30 13:44:11.714035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.713910 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" event={"ID":"2ac2db76-8e24-4899-aa7c-71755b76ead6","Type":"ContainerStarted","Data":"ee59f77bf0599f847473e69d107f67263dbd872e0a84040865430696b7d2510f"} Oct 30 13:44:11.717711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.716269 5049 generic.go:356] "Generic (PLEG): container finished" podID="6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e" containerID="874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08" exitCode=0 Oct 30 13:44:11.717711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.716341 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" event={"ID":"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e","Type":"ContainerDied","Data":"874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08"} Oct 30 13:44:11.717711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.716374 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" event={"ID":"6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e","Type":"ContainerDied","Data":"df59ca7d0631a41d5f68f72bcd66ae5f6c9a990c1857aa2c6fd3e79b6764548c"} Oct 30 13:44:11.717711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.716424 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h" Oct 30 13:44:11.726903 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.726872 5049 scope.go:122] "RemoveContainer" containerID="66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c" Oct 30 13:44:11.727189 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:11.727170 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c\": container with ID starting with 66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c not found: ID does not exist" containerID="66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c" Oct 30 13:44:11.727282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.727199 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c"} err="failed to get container status \"66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c\": rpc error: code = NotFound desc = could not find container \"66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c\": container with ID starting with 66665322434f324a504f774d31d93b90d1bb1a343ab23eb89fc38ce02f79402c not found: ID does not exist" Oct 30 13:44:11.727282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.727222 5049 scope.go:122] "RemoveContainer" containerID="874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08" Oct 30 13:44:11.737675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.737655 5049 scope.go:122] "RemoveContainer" containerID="874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08" Oct 30 13:44:11.737954 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:11.737938 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08\": container with ID starting with 874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08 not found: ID does not exist" containerID="874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08" Oct 30 13:44:11.738007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.737960 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08"} err="failed to get container status \"874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08\": rpc error: code = NotFound desc = could not find container \"874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08\": container with ID starting with 874cf38f8a0b304ddc092ac511f97f44ea335fe02e767f98f2503355a8875d08 not found: ID does not exist" Oct 30 13:44:11.765399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.765313 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" podStartSLOduration=1.765298053 podStartE2EDuration="1.765298053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:11.763299633 +0000 UTC m=+3158.578494056" watchObservedRunningTime="2026-10-30 13:44:11.765298053 +0000 UTC m=+3158.580492476" Oct 30 13:44:11.768159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.768130 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbc92\" (UniqueName: \"kubernetes.io/projected/22699b7b-4d79-427b-a469-a8d2d87a7e51-kube-api-access-cbc92\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:11.768159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.768155 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzcqg\" (UniqueName: \"kubernetes.io/projected/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e-kube-api-access-lzcqg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:11.810361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.810314 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb"] Oct 30 13:44:11.818236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.818212 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9cbcbc-6fd795bcf-zwvgb"] Oct 30 13:44:11.847478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.847452 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h"] Oct 30 13:44:11.850555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:11.850530 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9cbcbc-b98dddc86-tqt7h"] Oct 30 13:44:12.287046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:12.287016 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:12.478986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:12.478950 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:12.733709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:12.733652 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:12.748431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:12.748396 5049 generic.go:356] "Generic (PLEG): container finished" podID="2ac2db76-8e24-4899-aa7c-71755b76ead6" containerID="9691f7708a2f46f6d45527fbcbfc4fd10111d487f13b3617bea37a5cae454b71" exitCode=0 Oct 30 13:44:12.748621 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:12.748482 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" event={"ID":"2ac2db76-8e24-4899-aa7c-71755b76ead6","Type":"ContainerDied","Data":"9691f7708a2f46f6d45527fbcbfc4fd10111d487f13b3617bea37a5cae454b71"} Oct 30 13:44:13.179098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:13.179069 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:13.286011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:13.285976 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb"] Oct 30 13:44:13.291571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:13.291545 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fca422-d62rb"] Oct 30 13:44:13.336105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:13.336073 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22699b7b-4d79-427b-a469-a8d2d87a7e51" path="/var/lib/kubelet/pods/22699b7b-4d79-427b-a469-a8d2d87a7e51/volumes" Oct 30 13:44:13.336585 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:13.336570 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1e5505-dd44-45c7-9745-ebf8052504a0" path="/var/lib/kubelet/pods/3c1e5505-dd44-45c7-9745-ebf8052504a0/volumes" Oct 30 13:44:13.336956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:13.336943 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e" path="/var/lib/kubelet/pods/6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e/volumes" Oct 30 13:44:13.376648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:13.376620 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:13.851865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:13.851829 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:13.918958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:13.918929 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:14.018459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.018370 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-openshift-ca\") pod \"2ac2db76-8e24-4899-aa7c-71755b76ead6\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " Oct 30 13:44:14.018619 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.018510 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac2db76-8e24-4899-aa7c-71755b76ead6-kube-api-access-pbb8m\" (UniqueName: \"kubernetes.io/projected/2ac2db76-8e24-4899-aa7c-71755b76ead6-kube-api-access-pbb8m\") pod \"2ac2db76-8e24-4899-aa7c-71755b76ead6\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " Oct 30 13:44:14.018711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.018677 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-xks-ca\") pod \"2ac2db76-8e24-4899-aa7c-71755b76ead6\" (UID: \"2ac2db76-8e24-4899-aa7c-71755b76ead6\") " Oct 30 13:44:14.018806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.018783 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-openshift-ca" pod "2ac2db76-8e24-4899-aa7c-71755b76ead6" (UID: "2ac2db76-8e24-4899-aa7c-71755b76ead6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:14.019107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.019087 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:14.020690 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.020663 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac2db76-8e24-4899-aa7c-71755b76ead6-kube-api-access-pbb8m" pod "2ac2db76-8e24-4899-aa7c-71755b76ead6" (UID: "2ac2db76-8e24-4899-aa7c-71755b76ead6"). InnerVolumeSpecName "kube-api-access-pbb8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:14.020779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.020671 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-xks-ca" pod "2ac2db76-8e24-4899-aa7c-71755b76ead6" (UID: "2ac2db76-8e24-4899-aa7c-71755b76ead6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:14.119913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.119877 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbb8m\" (UniqueName: \"kubernetes.io/projected/2ac2db76-8e24-4899-aa7c-71755b76ead6-kube-api-access-pbb8m\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:14.119913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.119911 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac2db76-8e24-4899-aa7c-71755b76ead6-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:14.770603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.770560 5049 generic.go:356] "Generic (PLEG): container finished" podID="8111c92e-8da9-49eb-ac5f-14427435e155" containerID="4d2c5f4703de51275fdb6aa5e549b87e61acd70e0ac016bf208674524b3ceb0a" exitCode=0 Oct 30 13:44:14.770902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.770873 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" event={"ID":"8111c92e-8da9-49eb-ac5f-14427435e155","Type":"ContainerDied","Data":"4d2c5f4703de51275fdb6aa5e549b87e61acd70e0ac016bf208674524b3ceb0a"} Oct 30 13:44:14.772667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.772634 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" event={"ID":"2ac2db76-8e24-4899-aa7c-71755b76ead6","Type":"ContainerDied","Data":"ee59f77bf0599f847473e69d107f67263dbd872e0a84040865430696b7d2510f"} Oct 30 13:44:14.772667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.772659 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc" Oct 30 13:44:14.772831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.772691 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee59f77bf0599f847473e69d107f67263dbd872e0a84040865430696b7d2510f" Oct 30 13:44:14.824221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.824195 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:44:14.927563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.927504 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-credential-socket\") pod \"8111c92e-8da9-49eb-ac5f-14427435e155\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " Oct 30 13:44:14.928049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.927597 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-istio-token\" (UniqueName: \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-istio-token\") pod \"8111c92e-8da9-49eb-ac5f-14427435e155\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " Oct 30 13:44:14.928049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.927639 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-certs\") pod \"8111c92e-8da9-49eb-ac5f-14427435e155\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " Oct 30 13:44:14.928049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.927683 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-data\") pod \"8111c92e-8da9-49eb-ac5f-14427435e155\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " Oct 30 13:44:14.928049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.927775 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-envoy\") pod \"8111c92e-8da9-49eb-ac5f-14427435e155\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " Oct 30 13:44:14.928049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.927857 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8111c92e-8da9-49eb-ac5f-14427435e155-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8111c92e-8da9-49eb-ac5f-14427435e155-istiod-ca-cert\") pod \"8111c92e-8da9-49eb-ac5f-14427435e155\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " Oct 30 13:44:14.928049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.927911 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-socket\") pod \"8111c92e-8da9-49eb-ac5f-14427435e155\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " Oct 30 13:44:14.928049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.927969 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8111c92e-8da9-49eb-ac5f-14427435e155-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8111c92e-8da9-49eb-ac5f-14427435e155-istio-podinfo\") pod \"8111c92e-8da9-49eb-ac5f-14427435e155\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " Oct 30 13:44:14.928049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.928050 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-kube-api-access-9rwq8\" (UniqueName: \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-kube-api-access-9rwq8\") pod \"8111c92e-8da9-49eb-ac5f-14427435e155\" (UID: \"8111c92e-8da9-49eb-ac5f-14427435e155\") " Oct 30 13:44:14.928658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.928632 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-data" pod "8111c92e-8da9-49eb-ac5f-14427435e155" (UID: "8111c92e-8da9-49eb-ac5f-14427435e155"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:14.928921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.928889 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-credential-socket" pod "8111c92e-8da9-49eb-ac5f-14427435e155" (UID: "8111c92e-8da9-49eb-ac5f-14427435e155"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:14.929061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.929004 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8111c92e-8da9-49eb-ac5f-14427435e155-istiod-ca-cert" pod "8111c92e-8da9-49eb-ac5f-14427435e155" (UID: "8111c92e-8da9-49eb-ac5f-14427435e155"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:14.929259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.929233 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-certs" pod "8111c92e-8da9-49eb-ac5f-14427435e155" (UID: "8111c92e-8da9-49eb-ac5f-14427435e155"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:14.929426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.929392 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-socket" pod "8111c92e-8da9-49eb-ac5f-14427435e155" (UID: "8111c92e-8da9-49eb-ac5f-14427435e155"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:14.931403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.931373 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-kube-api-access-9rwq8" pod "8111c92e-8da9-49eb-ac5f-14427435e155" (UID: "8111c92e-8da9-49eb-ac5f-14427435e155"). InnerVolumeSpecName "kube-api-access-9rwq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:14.931606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.931579 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8111c92e-8da9-49eb-ac5f-14427435e155-istio-podinfo" pod "8111c92e-8da9-49eb-ac5f-14427435e155" (UID: "8111c92e-8da9-49eb-ac5f-14427435e155"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:14.931606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.931593 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-istio-token" pod "8111c92e-8da9-49eb-ac5f-14427435e155" (UID: "8111c92e-8da9-49eb-ac5f-14427435e155"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:14.931752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:14.931658 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-envoy" pod "8111c92e-8da9-49eb-ac5f-14427435e155" (UID: "8111c92e-8da9-49eb-ac5f-14427435e155"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:15.030245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.030205 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.030245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.030243 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8111c92e-8da9-49eb-ac5f-14427435e155-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.030245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.030253 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.030602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.030261 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8111c92e-8da9-49eb-ac5f-14427435e155-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.030602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.030271 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rwq8\" (UniqueName: \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-kube-api-access-9rwq8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.030602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.030280 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.030602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.030288 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8111c92e-8da9-49eb-ac5f-14427435e155-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.030602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.030296 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.030602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.030303 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8111c92e-8da9-49eb-ac5f-14427435e155-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.182999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.182918 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:15.363891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.363859 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:15.781337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.781293 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" Oct 30 13:44:15.781545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.781344 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w" event={"ID":"8111c92e-8da9-49eb-ac5f-14427435e155","Type":"ContainerDied","Data":"e2645ad10e55145033fc7d4446da30080f50751d0f8d4a6c631c24339cd30931"} Oct 30 13:44:15.781545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.781404 5049 scope.go:122] "RemoveContainer" containerID="4d2c5f4703de51275fdb6aa5e549b87e61acd70e0ac016bf208674524b3ceb0a" Oct 30 13:44:15.849053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.849021 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w"] Oct 30 13:44:15.860600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:15.860572 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-x5m6w"] Oct 30 13:44:16.178933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:16.178901 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:16.255065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:16.255025 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5"] Oct 30 13:44:16.258420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:16.258393 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fdd4d4-zx7h5"] Oct 30 13:44:16.360044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:16.360013 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:16.845467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:16.845435 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:17.337445 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:17.337405 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aaf5eb0-8e93-4671-93ab-34d20e197783" path="/var/lib/kubelet/pods/2aaf5eb0-8e93-4671-93ab-34d20e197783/volumes" Oct 30 13:44:17.338422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:17.338397 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8111c92e-8da9-49eb-ac5f-14427435e155" path="/var/lib/kubelet/pods/8111c92e-8da9-49eb-ac5f-14427435e155/volumes" Oct 30 13:44:17.417302 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:17.417264 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:17.621533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:17.621436 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:18.042384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:18.042291 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:18.289646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:18.289610 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:18.495427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:18.495387 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:18.907831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:18.907795 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:19.096107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.096073 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:19.138144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.138099 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh"] Oct 30 13:44:19.148228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.148189 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ff4285-6pwfh"] Oct 30 13:44:19.227570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.227492 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc"] Oct 30 13:44:19.228393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228373 5049 cpu_manager.go:414] "containerMap: removing container" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" Oct 30 13:44:19.228440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228396 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" Oct 30 13:44:19.228440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228425 5049 cpu_manager.go:414] "containerMap: removing container" podUID="8111c92e-8da9-49eb-ac5f-14427435e155" containerName="istio-proxy" Oct 30 13:44:19.228440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228434 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8111c92e-8da9-49eb-ac5f-14427435e155" containerName="istio-proxy" Oct 30 13:44:19.228555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228456 5049 cpu_manager.go:414] "containerMap: removing container" podUID="22699b7b-4d79-427b-a469-a8d2d87a7e51" containerName="payload-pre-processing" Oct 30 13:44:19.228555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228466 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22699b7b-4d79-427b-a469-a8d2d87a7e51" containerName="payload-pre-processing" Oct 30 13:44:19.228555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228488 5049 cpu_manager.go:414] "containerMap: removing container" podUID="6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e" containerName="payload-processing" Oct 30 13:44:19.228555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228496 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e" containerName="payload-processing" Oct 30 13:44:19.228555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228528 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2ac2db76-8e24-4899-aa7c-71755b76ead6" containerName="revoke-keys" Oct 30 13:44:19.228555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228535 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac2db76-8e24-4899-aa7c-71755b76ead6" containerName="revoke-keys" Oct 30 13:44:19.228740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228668 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac2db76-8e24-4899-aa7c-71755b76ead6" containerName="revoke-keys" Oct 30 13:44:19.228740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228685 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ab940e-48f1-4439-8842-e7bbba2b45b7" containerName="istio-proxy" Oct 30 13:44:19.228740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228699 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="8111c92e-8da9-49eb-ac5f-14427435e155" containerName="istio-proxy" Oct 30 13:44:19.228740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228709 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="22699b7b-4d79-427b-a469-a8d2d87a7e51" containerName="payload-pre-processing" Oct 30 13:44:19.228740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.228722 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9e2dd1-1a8f-4ed8-ab27-94c06522f93e" containerName="payload-processing" Oct 30 13:44:19.234234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.234215 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.247271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.247245 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:44:19.247435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.247288 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:44:19.263947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.263913 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc"] Oct 30 13:44:19.312950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.312917 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46zd\" (UniqueName: \"kubernetes.io/projected/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-kube-api-access-f46zd\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.313127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.312967 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.313180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.313146 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.337008 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.336973 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a71238-c862-4567-96ab-57bec5c3840e" path="/var/lib/kubelet/pods/77a71238-c862-4567-96ab-57bec5c3840e/volumes" Oct 30 13:44:19.414700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.414664 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46zd\" (UniqueName: \"kubernetes.io/projected/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-kube-api-access-f46zd\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.414924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.414724 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.414924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.414850 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.415563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.415539 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.417406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.417378 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.438389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.438353 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46zd\" (UniqueName: \"kubernetes.io/projected/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-kube-api-access-f46zd\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.551851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.551817 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:19.552970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.552941 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:19.860116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:19.860093 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc"] Oct 30 13:44:19.863465 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:44:19.863434 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9250b2b2_0fda_4ac9_afdd_1efc4819bba2.slice/crio-799aaf50c0070c596e06a387987a3f62253a7d9aeddce7eb26917822b292df34 WatchSource:0}: Error finding container 799aaf50c0070c596e06a387987a3f62253a7d9aeddce7eb26917822b292df34: Status 404 returned error can't find the container with id 799aaf50c0070c596e06a387987a3f62253a7d9aeddce7eb26917822b292df34 Oct 30 13:44:20.602178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:20.602140 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk"] Oct 30 13:44:20.618647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:20.618610 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr"] Oct 30 13:44:20.825436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:20.825395 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:20.826148 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:20.826059 5049 generic.go:356] "Generic (PLEG): container finished" podID="9250b2b2-0fda-4ac9-afdd-1efc4819bba2" containerID="985f44949db928439eaed7df39d65d2ae55ac6a556529f46f55799319511585d" exitCode=0 Oct 30 13:44:20.827668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:20.826247 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" event={"ID":"9250b2b2-0fda-4ac9-afdd-1efc4819bba2","Type":"ContainerDied","Data":"985f44949db928439eaed7df39d65d2ae55ac6a556529f46f55799319511585d"} Oct 30 13:44:20.827668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:20.826385 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" event={"ID":"9250b2b2-0fda-4ac9-afdd-1efc4819bba2","Type":"ContainerStarted","Data":"799aaf50c0070c596e06a387987a3f62253a7d9aeddce7eb26917822b292df34"} Oct 30 13:44:21.062221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:21.062181 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:21.347405 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:21.347285 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:21.818345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:21.813581 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:21.976111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:21.976069 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:22.078429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.066618 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx"] Oct 30 13:44:22.078429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.066842 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" containerID="cri-o://40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e" gracePeriod=30 Oct 30 13:44:22.078429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.073895 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:22.197097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.197061 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-openshift-ca\") pod \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " Oct 30 13:44:22.197480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.197449 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-xks-ca\") pod \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " Oct 30 13:44:22.197644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.197624 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-kube-api-access-f46zd\" (UniqueName: \"kubernetes.io/projected/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-kube-api-access-f46zd\") pod \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\" (UID: \"9250b2b2-0fda-4ac9-afdd-1efc4819bba2\") " Oct 30 13:44:22.199167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.199140 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-openshift-ca" pod "9250b2b2-0fda-4ac9-afdd-1efc4819bba2" (UID: "9250b2b2-0fda-4ac9-afdd-1efc4819bba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:22.230353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.224697 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-kube-api-access-f46zd" pod "9250b2b2-0fda-4ac9-afdd-1efc4819bba2" (UID: "9250b2b2-0fda-4ac9-afdd-1efc4819bba2"). InnerVolumeSpecName "kube-api-access-f46zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:22.230353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.216425 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:22.231797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.231763 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-xks-ca" pod "9250b2b2-0fda-4ac9-afdd-1efc4819bba2" (UID: "9250b2b2-0fda-4ac9-afdd-1efc4819bba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:22.300109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.300073 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:22.300109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.300111 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:22.300314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.300128 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f46zd\" (UniqueName: \"kubernetes.io/projected/9250b2b2-0fda-4ac9-afdd-1efc4819bba2-kube-api-access-f46zd\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:22.600001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.599957 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv"] Oct 30 13:44:22.600793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.600772 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9250b2b2-0fda-4ac9-afdd-1efc4819bba2" containerName="revoke-keys" Oct 30 13:44:22.600793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.600793 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9250b2b2-0fda-4ac9-afdd-1efc4819bba2" containerName="revoke-keys" Oct 30 13:44:22.600963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.600879 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9250b2b2-0fda-4ac9-afdd-1efc4819bba2" containerName="revoke-keys" Oct 30 13:44:22.604435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.604411 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:22.626852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.626819 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv"] Oct 30 13:44:22.704851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.704813 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:22.705003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.704872 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gpfg\" (UniqueName: \"kubernetes.io/projected/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-kube-api-access-8gpfg\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:22.705003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.704977 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:22.817272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.817221 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:22.817513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.817335 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gpfg\" (UniqueName: \"kubernetes.io/projected/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-kube-api-access-8gpfg\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:22.817513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.817399 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:22.818393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.818366 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:22.822477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.822451 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:22.844985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.844922 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gpfg\" (UniqueName: \"kubernetes.io/projected/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-kube-api-access-8gpfg\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:22.862962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.862887 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" event={"ID":"9250b2b2-0fda-4ac9-afdd-1efc4819bba2","Type":"ContainerDied","Data":"799aaf50c0070c596e06a387987a3f62253a7d9aeddce7eb26917822b292df34"} Oct 30 13:44:22.862962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.862951 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="799aaf50c0070c596e06a387987a3f62253a7d9aeddce7eb26917822b292df34" Oct 30 13:44:22.862962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.862900 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc" Oct 30 13:44:22.921976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:22.921933 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:23.228737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:23.228636 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv"] Oct 30 13:44:23.232133 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:44:23.232097 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3ab600_2ed6_40aa_8bb4_cef6afbee938.slice/crio-64a5b78765aada7d0c01c7a2d552352ba9e288aa80d1861fd29b47c876b6e40f WatchSource:0}: Error finding container 64a5b78765aada7d0c01c7a2d552352ba9e288aa80d1861fd29b47c876b6e40f: Status 404 returned error can't find the container with id 64a5b78765aada7d0c01c7a2d552352ba9e288aa80d1861fd29b47c876b6e40f Oct 30 13:44:23.454340 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:23.453389 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:23.873507 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:23.873473 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" event={"ID":"9c3ab600-2ed6-40aa-8bb4-cef6afbee938","Type":"ContainerStarted","Data":"b230a5e8357cdd3a7d2f0c76ed92b80941a81fc9d0e0da1e45c5dbceab112b86"} Oct 30 13:44:23.873924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:23.873521 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" event={"ID":"9c3ab600-2ed6-40aa-8bb4-cef6afbee938","Type":"ContainerStarted","Data":"64a5b78765aada7d0c01c7a2d552352ba9e288aa80d1861fd29b47c876b6e40f"} Oct 30 13:44:23.933881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:23.933818 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" podStartSLOduration=1.933802078 podStartE2EDuration="1.933802078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:23.932976629 +0000 UTC m=+3170.748171053" watchObservedRunningTime="2026-10-30 13:44:23.933802078 +0000 UTC m=+3170.748996501" Oct 30 13:44:24.805337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:24.805285 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:24.882871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:24.882839 5049 generic.go:356] "Generic (PLEG): container finished" podID="9c3ab600-2ed6-40aa-8bb4-cef6afbee938" containerID="b230a5e8357cdd3a7d2f0c76ed92b80941a81fc9d0e0da1e45c5dbceab112b86" exitCode=0 Oct 30 13:44:24.883384 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:24.882961 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" event={"ID":"9c3ab600-2ed6-40aa-8bb4-cef6afbee938","Type":"ContainerDied","Data":"b230a5e8357cdd3a7d2f0c76ed92b80941a81fc9d0e0da1e45c5dbceab112b86"} Oct 30 13:44:24.981485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:24.981451 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:25.185408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.177996 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:25.572368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.572305 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb"] Oct 30 13:44:25.576563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.576539 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:25.603981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.603939 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb"] Oct 30 13:44:25.609615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.609575 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:25.619803 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.619765 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" podUID="14634916-26ab-4c10-bfba-03002f7a4eab" containerName="payload-pre-processing" containerID="cri-o://9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c" gracePeriod=25 Oct 30 13:44:25.766400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.766366 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:25.766584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.766559 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:25.766644 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.766624 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjfj\" (UniqueName: \"kubernetes.io/projected/f3974584-4132-47aa-81c4-a988d90789d4-kube-api-access-ffjfj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:25.826622 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.826506 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" podUID="561552f7-0505-4fda-9fd1-eb1fa882a262" containerName="payload-processing" containerID="cri-o://526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4" gracePeriod=25 Oct 30 13:44:25.868272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.868238 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:25.868460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.868407 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjfj\" (UniqueName: \"kubernetes.io/projected/f3974584-4132-47aa-81c4-a988d90789d4-kube-api-access-ffjfj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:25.868583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.868568 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:25.869168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.869147 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:25.870658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.870636 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:25.891364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:25.891343 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjfj\" (UniqueName: \"kubernetes.io/projected/f3974584-4132-47aa-81c4-a988d90789d4-kube-api-access-ffjfj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:26.072999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.072973 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:26.172152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.172019 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-kube-api-access-8gpfg\" (UniqueName: \"kubernetes.io/projected/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-kube-api-access-8gpfg\") pod \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " Oct 30 13:44:26.172152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.172062 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-openshift-ca\") pod \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " Oct 30 13:44:26.172410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.172252 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-xks-ca\") pod \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\" (UID: \"9c3ab600-2ed6-40aa-8bb4-cef6afbee938\") " Oct 30 13:44:26.172850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.172826 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-openshift-ca" pod "9c3ab600-2ed6-40aa-8bb4-cef6afbee938" (UID: "9c3ab600-2ed6-40aa-8bb4-cef6afbee938"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:26.174380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.174359 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-kube-api-access-8gpfg" pod "9c3ab600-2ed6-40aa-8bb4-cef6afbee938" (UID: "9c3ab600-2ed6-40aa-8bb4-cef6afbee938"). InnerVolumeSpecName "kube-api-access-8gpfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:26.174471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.174436 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-xks-ca" pod "9c3ab600-2ed6-40aa-8bb4-cef6afbee938" (UID: "9c3ab600-2ed6-40aa-8bb4-cef6afbee938"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:26.191357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.191307 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:26.274396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.274360 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:26.274548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.274400 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gpfg\" (UniqueName: \"kubernetes.io/projected/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-kube-api-access-8gpfg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:26.274548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.274415 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ab600-2ed6-40aa-8bb4-cef6afbee938-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:26.519890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.519801 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb"] Oct 30 13:44:26.529658 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:44:26.529619 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3974584_4132_47aa_81c4_a988d90789d4.slice/crio-2300dd08c7bb0198e86da607904f3214a131b0b3450fe51b7f24c0207dd449e1 WatchSource:0}: Error finding container 2300dd08c7bb0198e86da607904f3214a131b0b3450fe51b7f24c0207dd449e1: Status 404 returned error can't find the container with id 2300dd08c7bb0198e86da607904f3214a131b0b3450fe51b7f24c0207dd449e1 Oct 30 13:44:26.679003 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:26.678961 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:44:26.726309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.726280 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" podUID="561552f7-0505-4fda-9fd1-eb1fa882a262" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:26.899728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.899696 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" event={"ID":"9c3ab600-2ed6-40aa-8bb4-cef6afbee938","Type":"ContainerDied","Data":"64a5b78765aada7d0c01c7a2d552352ba9e288aa80d1861fd29b47c876b6e40f"} Oct 30 13:44:26.899728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.899713 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv" Oct 30 13:44:26.900213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.899742 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64a5b78765aada7d0c01c7a2d552352ba9e288aa80d1861fd29b47c876b6e40f" Oct 30 13:44:26.901444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.901420 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" event={"ID":"f3974584-4132-47aa-81c4-a988d90789d4","Type":"ContainerStarted","Data":"2ca4088d7b8c5f4f45eae96fadf32eb3f0c80e2323ab5fd9152e0134a3498595"} Oct 30 13:44:26.901564 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.901460 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" event={"ID":"f3974584-4132-47aa-81c4-a988d90789d4","Type":"ContainerStarted","Data":"2300dd08c7bb0198e86da607904f3214a131b0b3450fe51b7f24c0207dd449e1"} Oct 30 13:44:26.941941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.941885 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" podStartSLOduration=1.941872283 podStartE2EDuration="1.941872283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:26.938460362 +0000 UTC m=+3173.753654786" watchObservedRunningTime="2026-10-30 13:44:26.941872283 +0000 UTC m=+3173.757066864" Oct 30 13:44:26.948179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.948145 5049 patch_prober.go:29] interesting pod/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:44:26.948395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:26.948203 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:44:27.373426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.373401 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:44:27.486229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486201 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-certs\") pod \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " Oct 30 13:44:27.486396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486241 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istiod-ca-cert\") pod \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " Oct 30 13:44:27.486396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486282 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-envoy\") pod \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " Oct 30 13:44:27.486529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486432 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-data\") pod \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " Oct 30 13:44:27.486529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486461 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-socket\") pod \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " Oct 30 13:44:27.486529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486488 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-token\" (UniqueName: \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-token\") pod \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " Oct 30 13:44:27.486529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486489 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-certs" pod "574a0063-5e8f-448f-b5d2-a2f182fdbc25" (UID: "574a0063-5e8f-448f-b5d2-a2f182fdbc25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:27.486765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486536 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-kube-api-access-skc47\" (UniqueName: \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-kube-api-access-skc47\") pod \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " Oct 30 13:44:27.486765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486565 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-credential-socket\") pod \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " Oct 30 13:44:27.486765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486703 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-podinfo\") pod \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\" (UID: \"574a0063-5e8f-448f-b5d2-a2f182fdbc25\") " Oct 30 13:44:27.486765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486723 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-socket" pod "574a0063-5e8f-448f-b5d2-a2f182fdbc25" (UID: "574a0063-5e8f-448f-b5d2-a2f182fdbc25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:27.486765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486739 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-data" pod "574a0063-5e8f-448f-b5d2-a2f182fdbc25" (UID: "574a0063-5e8f-448f-b5d2-a2f182fdbc25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:27.487039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.486759 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istiod-ca-cert" pod "574a0063-5e8f-448f-b5d2-a2f182fdbc25" (UID: "574a0063-5e8f-448f-b5d2-a2f182fdbc25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:27.487039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.487027 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-credential-socket" pod "574a0063-5e8f-448f-b5d2-a2f182fdbc25" (UID: "574a0063-5e8f-448f-b5d2-a2f182fdbc25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:27.487381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.487356 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.487483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.487384 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.487483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.487396 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.487483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.487404 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.487483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.487416 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.488902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.488878 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-token" pod "574a0063-5e8f-448f-b5d2-a2f182fdbc25" (UID: "574a0063-5e8f-448f-b5d2-a2f182fdbc25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:27.489012 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.488918 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-envoy" pod "574a0063-5e8f-448f-b5d2-a2f182fdbc25" (UID: "574a0063-5e8f-448f-b5d2-a2f182fdbc25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:27.489137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.489116 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-podinfo" pod "574a0063-5e8f-448f-b5d2-a2f182fdbc25" (UID: "574a0063-5e8f-448f-b5d2-a2f182fdbc25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:27.489230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.489140 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-kube-api-access-skc47" pod "574a0063-5e8f-448f-b5d2-a2f182fdbc25" (UID: "574a0063-5e8f-448f-b5d2-a2f182fdbc25"). InnerVolumeSpecName "kube-api-access-skc47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:27.565539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.565490 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:27.588569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.588537 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.588569 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.588565 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skc47\" (UniqueName: \"kubernetes.io/projected/574a0063-5e8f-448f-b5d2-a2f182fdbc25-kube-api-access-skc47\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.588819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.588577 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.588819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.588590 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574a0063-5e8f-448f-b5d2-a2f182fdbc25-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:27.741853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.741762 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" podUID="14634916-26ab-4c10-bfba-03002f7a4eab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:27.742565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.742542 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:27.845151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.845111 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66"] Oct 30 13:44:27.845453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.845423 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" containerID="cri-o://ef462c19b7fe24cb1635f6d5158a069948648bccef6ac007825776801f798487" gracePeriod=30 Oct 30 13:44:27.852129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.850488 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2"] Oct 30 13:44:27.852129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.850708 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" podUID="715c24ba-f9e7-4389-ba41-59b7ea89e582" containerName="istio-proxy" containerID="cri-o://380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5" gracePeriod=30 Oct 30 13:44:27.910418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.910377 5049 generic.go:356] "Generic (PLEG): container finished" podID="f3974584-4132-47aa-81c4-a988d90789d4" containerID="2ca4088d7b8c5f4f45eae96fadf32eb3f0c80e2323ab5fd9152e0134a3498595" exitCode=0 Oct 30 13:44:27.910841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.910466 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" event={"ID":"f3974584-4132-47aa-81c4-a988d90789d4","Type":"ContainerDied","Data":"2ca4088d7b8c5f4f45eae96fadf32eb3f0c80e2323ab5fd9152e0134a3498595"} Oct 30 13:44:27.912538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.912513 5049 generic.go:356] "Generic (PLEG): container finished" podID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerID="40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e" exitCode=0 Oct 30 13:44:27.912668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.912554 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" Oct 30 13:44:27.912668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.912593 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" event={"ID":"574a0063-5e8f-448f-b5d2-a2f182fdbc25","Type":"ContainerDied","Data":"40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e"} Oct 30 13:44:27.912668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.912652 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx" event={"ID":"574a0063-5e8f-448f-b5d2-a2f182fdbc25","Type":"ContainerDied","Data":"6b0bced5f65c4cbe5e7f0ef30ee2e82f466eb6c2d400f39628c1a1c9ba581e8a"} Oct 30 13:44:27.912786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.912683 5049 scope.go:122] "RemoveContainer" containerID="40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e" Oct 30 13:44:27.925029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.925007 5049 scope.go:122] "RemoveContainer" containerID="2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8" Oct 30 13:44:27.949344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.948967 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:27.949344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.949179 5049 scope.go:122] "RemoveContainer" containerID="40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e" Oct 30 13:44:27.951376 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:27.949713 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e\": container with ID starting with 40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e not found: ID does not exist" containerID="40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e" Oct 30 13:44:27.951376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.949797 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e"} err="failed to get container status \"40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e\": rpc error: code = NotFound desc = could not find container \"40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e\": container with ID starting with 40554ce5ea5b5085ad3e60709a9fce8d951bed312f3c3d1d002a1eebbd316b7e not found: ID does not exist" Oct 30 13:44:27.951376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.949846 5049 scope.go:122] "RemoveContainer" containerID="2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8" Oct 30 13:44:27.951376 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:27.950556 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8\": container with ID starting with 2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8 not found: ID does not exist" containerID="2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8" Oct 30 13:44:27.951376 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:27.950608 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8"} err="failed to get container status \"2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8\": rpc error: code = NotFound desc = could not find container \"2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8\": container with ID starting with 2e881d2eae6a751ab54ede86e776afe27087639553d162633bd829b24d2f42d8 not found: ID does not exist" Oct 30 13:44:28.028968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.028883 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx"] Oct 30 13:44:28.033647 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.033600 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-fca422-openshift-default-566d4bc6df-2xcsx"] Oct 30 13:44:28.130344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.130297 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:28.348429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.348390 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4"] Oct 30 13:44:28.349264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.349244 5049 cpu_manager.go:414] "containerMap: removing container" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" Oct 30 13:44:28.349333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.349268 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" Oct 30 13:44:28.349333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.349307 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9c3ab600-2ed6-40aa-8bb4-cef6afbee938" containerName="revoke-keys" Oct 30 13:44:28.349333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.349314 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3ab600-2ed6-40aa-8bb4-cef6afbee938" containerName="revoke-keys" Oct 30 13:44:28.349520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.349503 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" Oct 30 13:44:28.349588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.349526 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3ab600-2ed6-40aa-8bb4-cef6afbee938" containerName="revoke-keys" Oct 30 13:44:28.349588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.349541 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" Oct 30 13:44:28.355226 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.355200 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:28.384181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.384149 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4"] Oct 30 13:44:28.408366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.405095 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8bz\" (UniqueName: \"kubernetes.io/projected/46911daa-69f0-4084-92c6-1a555be94f61-kube-api-access-jk8bz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:28.408366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.405186 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:28.408366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.405407 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:28.506818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.506785 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8bz\" (UniqueName: \"kubernetes.io/projected/46911daa-69f0-4084-92c6-1a555be94f61-kube-api-access-jk8bz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:28.507019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.506939 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:28.507122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.507103 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:28.507936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.507898 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:28.509633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.509609 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:28.529630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.529607 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8bz\" (UniqueName: \"kubernetes.io/projected/46911daa-69f0-4084-92c6-1a555be94f61-kube-api-access-jk8bz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:28.673889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:28.673799 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:29.056404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.056365 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4"] Oct 30 13:44:29.139526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.139492 5049 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:44:29.139718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.139551 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:44:29.203474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.203446 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:29.319418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.319381 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-openshift-ca\") pod \"f3974584-4132-47aa-81c4-a988d90789d4\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " Oct 30 13:44:29.319571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.319533 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-xks-ca\") pod \"f3974584-4132-47aa-81c4-a988d90789d4\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " Oct 30 13:44:29.319658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.319639 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3974584-4132-47aa-81c4-a988d90789d4-kube-api-access-ffjfj\" (UniqueName: \"kubernetes.io/projected/f3974584-4132-47aa-81c4-a988d90789d4-kube-api-access-ffjfj\") pod \"f3974584-4132-47aa-81c4-a988d90789d4\" (UID: \"f3974584-4132-47aa-81c4-a988d90789d4\") " Oct 30 13:44:29.319836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.319803 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-openshift-ca" pod "f3974584-4132-47aa-81c4-a988d90789d4" (UID: "f3974584-4132-47aa-81c4-a988d90789d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:29.320168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.320144 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:29.322506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.322473 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3974584-4132-47aa-81c4-a988d90789d4-kube-api-access-ffjfj" pod "f3974584-4132-47aa-81c4-a988d90789d4" (UID: "f3974584-4132-47aa-81c4-a988d90789d4"). InnerVolumeSpecName "kube-api-access-ffjfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:29.322807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.322784 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-xks-ca" pod "f3974584-4132-47aa-81c4-a988d90789d4" (UID: "f3974584-4132-47aa-81c4-a988d90789d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:29.340948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.340914 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" path="/var/lib/kubelet/pods/574a0063-5e8f-448f-b5d2-a2f182fdbc25/volumes" Oct 30 13:44:29.379611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.379575 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk"] Oct 30 13:44:29.418378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.418349 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v"] Oct 30 13:44:29.421729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.421699 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3974584-4132-47aa-81c4-a988d90789d4-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:29.421867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.421735 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffjfj\" (UniqueName: \"kubernetes.io/projected/f3974584-4132-47aa-81c4-a988d90789d4-kube-api-access-ffjfj\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:29.588692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.588658 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:29.755973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.755940 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:29.938192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.938096 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" event={"ID":"46911daa-69f0-4084-92c6-1a555be94f61","Type":"ContainerStarted","Data":"d0e0303e1b3ea25fb2675bed5e2c03628b48a7bdfad0e4e1b81adbb502c8940d"} Oct 30 13:44:29.938192 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.938172 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" event={"ID":"46911daa-69f0-4084-92c6-1a555be94f61","Type":"ContainerStarted","Data":"d34d501786b1e52b59919f7e1290e299dc7d901ded3862681b3417806463b865"} Oct 30 13:44:29.939820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.939795 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" Oct 30 13:44:29.939992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.939834 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb" event={"ID":"f3974584-4132-47aa-81c4-a988d90789d4","Type":"ContainerDied","Data":"2300dd08c7bb0198e86da607904f3214a131b0b3450fe51b7f24c0207dd449e1"} Oct 30 13:44:29.939992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.939873 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2300dd08c7bb0198e86da607904f3214a131b0b3450fe51b7f24c0207dd449e1" Oct 30 13:44:29.981187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:29.981116 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" podStartSLOduration=1.981095856 podStartE2EDuration="1.981095856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:29.976302405 +0000 UTC m=+3176.791496829" watchObservedRunningTime="2026-10-30 13:44:29.981095856 +0000 UTC m=+3176.796290281" Oct 30 13:44:30.121195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:30.121145 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:30.491230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:30.491197 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:30.949100 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:30.949067 5049 generic.go:356] "Generic (PLEG): container finished" podID="46911daa-69f0-4084-92c6-1a555be94f61" containerID="d0e0303e1b3ea25fb2675bed5e2c03628b48a7bdfad0e4e1b81adbb502c8940d" exitCode=0 Oct 30 13:44:30.949259 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:30.949147 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" event={"ID":"46911daa-69f0-4084-92c6-1a555be94f61","Type":"ContainerDied","Data":"d0e0303e1b3ea25fb2675bed5e2c03628b48a7bdfad0e4e1b81adbb502c8940d"} Oct 30 13:44:31.875256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.875222 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd"] Oct 30 13:44:31.875857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.875841 5049 cpu_manager.go:414] "containerMap: removing container" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" Oct 30 13:44:31.875913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.875859 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="574a0063-5e8f-448f-b5d2-a2f182fdbc25" containerName="istio-proxy" Oct 30 13:44:31.875913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.875898 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f3974584-4132-47aa-81c4-a988d90789d4" containerName="revoke-keys" Oct 30 13:44:31.875913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.875905 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3974584-4132-47aa-81c4-a988d90789d4" containerName="revoke-keys" Oct 30 13:44:31.876017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.876008 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3974584-4132-47aa-81c4-a988d90789d4" containerName="revoke-keys" Oct 30 13:44:31.879796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.879778 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:31.932504 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.932470 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs"] Oct 30 13:44:31.936675 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.936651 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:31.939503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.939479 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd"] Oct 30 13:44:31.956676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.956643 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wpjs\" (UniqueName: \"kubernetes.io/projected/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-kube-api-access-9wpjs\") pod \"maas-api-revoke-sub-1c2c6ae4-4ktfd\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:31.956816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.956688 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c2c6ae4-4ktfd\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:31.956898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.956875 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c2c6ae4-4ktfd\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:31.961722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:31.960867 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs"] Oct 30 13:44:32.058280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.058238 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c2c6ae4-4ktfd\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:32.058500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.058291 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0f26a78-27pzs\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:32.058500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.058490 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0f26a78-27pzs\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:32.058623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.058531 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wpjs\" (UniqueName: \"kubernetes.io/projected/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-kube-api-access-9wpjs\") pod \"maas-api-revoke-sub-1c2c6ae4-4ktfd\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:32.058623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.058558 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c2c6ae4-4ktfd\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:32.058731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.058622 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6s6z\" (UniqueName: \"kubernetes.io/projected/41593460-aec9-43dd-a4ad-27be23c7b607-kube-api-access-j6s6z\") pod \"maas-api-revoke-sub-d0f26a78-27pzs\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:32.059457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.059425 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c2c6ae4-4ktfd\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:32.061237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.061214 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c2c6ae4-4ktfd\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:32.082090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.082055 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wpjs\" (UniqueName: \"kubernetes.io/projected/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-kube-api-access-9wpjs\") pod \"maas-api-revoke-sub-1c2c6ae4-4ktfd\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:32.134387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.134305 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:32.159934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.159904 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0f26a78-27pzs\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:32.160107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.159968 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6s6z\" (UniqueName: \"kubernetes.io/projected/41593460-aec9-43dd-a4ad-27be23c7b607-kube-api-access-j6s6z\") pod \"maas-api-revoke-sub-d0f26a78-27pzs\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:32.160107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.160020 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0f26a78-27pzs\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:32.160683 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.160656 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0f26a78-27pzs\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:32.162269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.162247 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0f26a78-27pzs\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:32.183454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.183426 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6s6z\" (UniqueName: \"kubernetes.io/projected/41593460-aec9-43dd-a4ad-27be23c7b607-kube-api-access-j6s6z\") pod \"maas-api-revoke-sub-d0f26a78-27pzs\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:32.194179 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.194148 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:32.257800 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.257740 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:32.261487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.261448 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46911daa-69f0-4084-92c6-1a555be94f61-kube-api-access-jk8bz\" (UniqueName: \"kubernetes.io/projected/46911daa-69f0-4084-92c6-1a555be94f61-kube-api-access-jk8bz\") pod \"46911daa-69f0-4084-92c6-1a555be94f61\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " Oct 30 13:44:32.261823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.261804 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-openshift-ca\") pod \"46911daa-69f0-4084-92c6-1a555be94f61\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " Oct 30 13:44:32.261909 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.261895 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-xks-ca\") pod \"46911daa-69f0-4084-92c6-1a555be94f61\" (UID: \"46911daa-69f0-4084-92c6-1a555be94f61\") " Oct 30 13:44:32.262239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.262211 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-openshift-ca" pod "46911daa-69f0-4084-92c6-1a555be94f61" (UID: "46911daa-69f0-4084-92c6-1a555be94f61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:32.262613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.262588 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:32.268819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.264415 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-xks-ca" pod "46911daa-69f0-4084-92c6-1a555be94f61" (UID: "46911daa-69f0-4084-92c6-1a555be94f61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:32.268819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.264493 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46911daa-69f0-4084-92c6-1a555be94f61-kube-api-access-jk8bz" pod "46911daa-69f0-4084-92c6-1a555be94f61" (UID: "46911daa-69f0-4084-92c6-1a555be94f61"). InnerVolumeSpecName "kube-api-access-jk8bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:32.365105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.364760 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk8bz\" (UniqueName: \"kubernetes.io/projected/46911daa-69f0-4084-92c6-1a555be94f61-kube-api-access-jk8bz\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:32.365105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.364800 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46911daa-69f0-4084-92c6-1a555be94f61-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:32.406408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.405472 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:32.512788 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.512761 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd"] Oct 30 13:44:32.514355 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:44:32.514313 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ce5b64_cf90_4d4e_84d3_f3d59a1eeef1.slice/crio-2a5339f613e2c8d6c557e0db3117c510382e9dde6768c2880fcb763e83212726 WatchSource:0}: Error finding container 2a5339f613e2c8d6c557e0db3117c510382e9dde6768c2880fcb763e83212726: Status 404 returned error can't find the container with id 2a5339f613e2c8d6c557e0db3117c510382e9dde6768c2880fcb763e83212726 Oct 30 13:44:32.621415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.621381 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs"] Oct 30 13:44:32.625429 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:44:32.625403 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41593460_aec9_43dd_a4ad_27be23c7b607.slice/crio-16310b184de8dab6db2b98c9fb3bbbca0e2e15f516c1765ef1dd3db0829463a7 WatchSource:0}: Error finding container 16310b184de8dab6db2b98c9fb3bbbca0e2e15f516c1765ef1dd3db0829463a7: Status 404 returned error can't find the container with id 16310b184de8dab6db2b98c9fb3bbbca0e2e15f516c1765ef1dd3db0829463a7 Oct 30 13:44:32.974444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.974414 5049 generic.go:356] "Generic (PLEG): container finished" podID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerID="ef462c19b7fe24cb1635f6d5158a069948648bccef6ac007825776801f798487" exitCode=0 Oct 30 13:44:32.974823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.974489 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" event={"ID":"71949dbf-4b7a-45e7-bd4f-775596a65fbd","Type":"ContainerDied","Data":"ef462c19b7fe24cb1635f6d5158a069948648bccef6ac007825776801f798487"} Oct 30 13:44:32.976510 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.976483 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" event={"ID":"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1","Type":"ContainerStarted","Data":"0d8f165e3f43c86fd26bc9ab20f0ca3f54c6d4366eb9b1e1e5687e5b11286f4c"} Oct 30 13:44:32.976651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.976526 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" event={"ID":"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1","Type":"ContainerStarted","Data":"2a5339f613e2c8d6c557e0db3117c510382e9dde6768c2880fcb763e83212726"} Oct 30 13:44:32.992089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.992020 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" Oct 30 13:44:32.992089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.992020 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4" event={"ID":"46911daa-69f0-4084-92c6-1a555be94f61","Type":"ContainerDied","Data":"d34d501786b1e52b59919f7e1290e299dc7d901ded3862681b3417806463b865"} Oct 30 13:44:32.992214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.992119 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d34d501786b1e52b59919f7e1290e299dc7d901ded3862681b3417806463b865" Oct 30 13:44:32.995371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.994793 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" event={"ID":"41593460-aec9-43dd-a4ad-27be23c7b607","Type":"ContainerStarted","Data":"bf67a35dde3b4a64de714b746b018b7ba39a6750d47e961f1a288a4a947b3e0e"} Oct 30 13:44:32.995371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:32.994845 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" event={"ID":"41593460-aec9-43dd-a4ad-27be23c7b607","Type":"ContainerStarted","Data":"16310b184de8dab6db2b98c9fb3bbbca0e2e15f516c1765ef1dd3db0829463a7"} Oct 30 13:44:33.023751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.023697 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" podStartSLOduration=2.023683018 podStartE2EDuration="2.023683018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:33.021128829 +0000 UTC m=+3179.836323250" watchObservedRunningTime="2026-10-30 13:44:33.023683018 +0000 UTC m=+3179.838877484" Oct 30 13:44:33.095265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.095215 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" podStartSLOduration=2.09520014 podStartE2EDuration="2.09520014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:33.091946356 +0000 UTC m=+3179.907140780" watchObservedRunningTime="2026-10-30 13:44:33.09520014 +0000 UTC m=+3179.910394556" Oct 30 13:44:33.160314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.160281 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:44:33.225965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.225935 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:44:33.269871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.269832 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:33.278230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278205 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-token\" (UniqueName: \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-token\") pod \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " Oct 30 13:44:33.278379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278238 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istiod-ca-cert\") pod \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " Oct 30 13:44:33.278379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278270 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-credential-socket\") pod \"715c24ba-f9e7-4389-ba41-59b7ea89e582\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " Oct 30 13:44:33.278472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278402 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-data\") pod \"715c24ba-f9e7-4389-ba41-59b7ea89e582\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " Oct 30 13:44:33.278534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278494 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-socket\") pod \"715c24ba-f9e7-4389-ba41-59b7ea89e582\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " Oct 30 13:44:33.278593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278543 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715c24ba-f9e7-4389-ba41-59b7ea89e582-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715c24ba-f9e7-4389-ba41-59b7ea89e582-istiod-ca-cert\") pod \"715c24ba-f9e7-4389-ba41-59b7ea89e582\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " Oct 30 13:44:33.278593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278546 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-credential-socket" pod "715c24ba-f9e7-4389-ba41-59b7ea89e582" (UID: "715c24ba-f9e7-4389-ba41-59b7ea89e582"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.278593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278574 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-podinfo\") pod \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " Oct 30 13:44:33.278784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278654 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-data" pod "715c24ba-f9e7-4389-ba41-59b7ea89e582" (UID: "715c24ba-f9e7-4389-ba41-59b7ea89e582"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.278784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278665 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istiod-ca-cert" pod "71949dbf-4b7a-45e7-bd4f-775596a65fbd" (UID: "71949dbf-4b7a-45e7-bd4f-775596a65fbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:33.278784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278658 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-kube-api-access-wwhcm\" (UniqueName: \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-kube-api-access-wwhcm\") pod \"715c24ba-f9e7-4389-ba41-59b7ea89e582\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " Oct 30 13:44:33.278784 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278709 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-envoy\") pod \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " Oct 30 13:44:33.279000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278788 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-token\" (UniqueName: \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-token\") pod \"715c24ba-f9e7-4389-ba41-59b7ea89e582\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " Oct 30 13:44:33.279000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278844 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-certs\") pod \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " Oct 30 13:44:33.279000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278891 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-credential-socket\") pod \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " Oct 30 13:44:33.279000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278950 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-socket\") pod \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " Oct 30 13:44:33.279219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.279014 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-certs\") pod \"715c24ba-f9e7-4389-ba41-59b7ea89e582\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " Oct 30 13:44:33.279219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.279044 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-envoy\") pod \"715c24ba-f9e7-4389-ba41-59b7ea89e582\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " Oct 30 13:44:33.279219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.279121 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-kube-api-access-p6w8q\" (UniqueName: \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-kube-api-access-p6w8q\") pod \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " Oct 30 13:44:33.279219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.279169 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-podinfo\") pod \"715c24ba-f9e7-4389-ba41-59b7ea89e582\" (UID: \"715c24ba-f9e7-4389-ba41-59b7ea89e582\") " Oct 30 13:44:33.279463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.279221 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-data\") pod \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\" (UID: \"71949dbf-4b7a-45e7-bd4f-775596a65fbd\") " Oct 30 13:44:33.279463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.278790 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-socket" pod "715c24ba-f9e7-4389-ba41-59b7ea89e582" (UID: "715c24ba-f9e7-4389-ba41-59b7ea89e582"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.279463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.279017 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715c24ba-f9e7-4389-ba41-59b7ea89e582-istiod-ca-cert" pod "715c24ba-f9e7-4389-ba41-59b7ea89e582" (UID: "715c24ba-f9e7-4389-ba41-59b7ea89e582"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:33.279658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.279563 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-certs" pod "71949dbf-4b7a-45e7-bd4f-775596a65fbd" (UID: "71949dbf-4b7a-45e7-bd4f-775596a65fbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.279981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.279789 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-certs" pod "715c24ba-f9e7-4389-ba41-59b7ea89e582" (UID: "715c24ba-f9e7-4389-ba41-59b7ea89e582"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.279981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.279791 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-data" pod "71949dbf-4b7a-45e7-bd4f-775596a65fbd" (UID: "71949dbf-4b7a-45e7-bd4f-775596a65fbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.280837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.280412 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.280837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.280442 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.280837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.280456 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.280837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.280468 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.280837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.280481 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.280837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.280496 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.280837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.280510 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.280837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.280523 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715c24ba-f9e7-4389-ba41-59b7ea89e582-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.281346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.281161 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-token" pod "71949dbf-4b7a-45e7-bd4f-775596a65fbd" (UID: "71949dbf-4b7a-45e7-bd4f-775596a65fbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:33.281506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.281448 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-credential-socket" pod "71949dbf-4b7a-45e7-bd4f-775596a65fbd" (UID: "71949dbf-4b7a-45e7-bd4f-775596a65fbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.281799 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.281724 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-socket" pod "71949dbf-4b7a-45e7-bd4f-775596a65fbd" (UID: "71949dbf-4b7a-45e7-bd4f-775596a65fbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.282365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.281909 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-envoy" pod "71949dbf-4b7a-45e7-bd4f-775596a65fbd" (UID: "71949dbf-4b7a-45e7-bd4f-775596a65fbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.282365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.282068 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-envoy" pod "715c24ba-f9e7-4389-ba41-59b7ea89e582" (UID: "715c24ba-f9e7-4389-ba41-59b7ea89e582"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.282365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.282064 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-podinfo" pod "71949dbf-4b7a-45e7-bd4f-775596a65fbd" (UID: "71949dbf-4b7a-45e7-bd4f-775596a65fbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:33.282365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.282113 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-kube-api-access-wwhcm" pod "715c24ba-f9e7-4389-ba41-59b7ea89e582" (UID: "715c24ba-f9e7-4389-ba41-59b7ea89e582"). InnerVolumeSpecName "kube-api-access-wwhcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:33.282365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.282311 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-token" pod "715c24ba-f9e7-4389-ba41-59b7ea89e582" (UID: "715c24ba-f9e7-4389-ba41-59b7ea89e582"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:33.282673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.282561 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-kube-api-access-p6w8q" pod "71949dbf-4b7a-45e7-bd4f-775596a65fbd" (UID: "71949dbf-4b7a-45e7-bd4f-775596a65fbd"). InnerVolumeSpecName "kube-api-access-p6w8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:33.283307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.283289 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-podinfo" pod "715c24ba-f9e7-4389-ba41-59b7ea89e582" (UID: "715c24ba-f9e7-4389-ba41-59b7ea89e582"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:33.381215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.381182 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.381215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.381214 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6w8q\" (UniqueName: \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-kube-api-access-p6w8q\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.381215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.381224 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.381566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.381233 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.381566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.381241 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.381566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.381248 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhcm\" (UniqueName: \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-kube-api-access-wwhcm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.381566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.381257 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.381566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.381265 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715c24ba-f9e7-4389-ba41-59b7ea89e582-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.381566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.381273 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.381566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.381281 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71949dbf-4b7a-45e7-bd4f-775596a65fbd-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.457414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:33.457307 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:34.010605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.010554 5049 generic.go:356] "Generic (PLEG): container finished" podID="41593460-aec9-43dd-a4ad-27be23c7b607" containerID="bf67a35dde3b4a64de714b746b018b7ba39a6750d47e961f1a288a4a947b3e0e" exitCode=0 Oct 30 13:44:34.011075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.010743 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" event={"ID":"41593460-aec9-43dd-a4ad-27be23c7b607","Type":"ContainerDied","Data":"bf67a35dde3b4a64de714b746b018b7ba39a6750d47e961f1a288a4a947b3e0e"} Oct 30 13:44:34.013152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.013114 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" Oct 30 13:44:34.013834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.013809 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66" event={"ID":"71949dbf-4b7a-45e7-bd4f-775596a65fbd","Type":"ContainerDied","Data":"19f774031dd3d8300784991ce5957ad102dfa2f263f46e2f1e16c070bb1217dd"} Oct 30 13:44:34.013970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.013869 5049 scope.go:122] "RemoveContainer" containerID="ef462c19b7fe24cb1635f6d5158a069948648bccef6ac007825776801f798487" Oct 30 13:44:34.016721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.016686 5049 generic.go:356] "Generic (PLEG): container finished" podID="65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1" containerID="0d8f165e3f43c86fd26bc9ab20f0ca3f54c6d4366eb9b1e1e5687e5b11286f4c" exitCode=0 Oct 30 13:44:34.017055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.017027 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" event={"ID":"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1","Type":"ContainerDied","Data":"0d8f165e3f43c86fd26bc9ab20f0ca3f54c6d4366eb9b1e1e5687e5b11286f4c"} Oct 30 13:44:34.018761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.018734 5049 generic.go:356] "Generic (PLEG): container finished" podID="715c24ba-f9e7-4389-ba41-59b7ea89e582" containerID="380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5" exitCode=0 Oct 30 13:44:34.018882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.018789 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" Oct 30 13:44:34.018882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.018825 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" event={"ID":"715c24ba-f9e7-4389-ba41-59b7ea89e582","Type":"ContainerDied","Data":"380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5"} Oct 30 13:44:34.018882 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.018859 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2" event={"ID":"715c24ba-f9e7-4389-ba41-59b7ea89e582","Type":"ContainerDied","Data":"9871ad7aaa0aa319e0f9579b648a9f6b4a3baed904f3ae1f304a81ca5473f4ef"} Oct 30 13:44:34.029642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.029620 5049 scope.go:122] "RemoveContainer" containerID="380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5" Oct 30 13:44:34.043506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.043486 5049 scope.go:122] "RemoveContainer" containerID="380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5" Oct 30 13:44:34.043822 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:34.043797 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5\": container with ID starting with 380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5 not found: ID does not exist" containerID="380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5" Oct 30 13:44:34.043895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.043837 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5"} err="failed to get container status \"380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5\": rpc error: code = NotFound desc = could not find container \"380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5\": container with ID starting with 380e1326d81bd43da0b653417ab94ededd2bc8285019c0ec0ec21751ac7a66f5 not found: ID does not exist" Oct 30 13:44:34.102059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.102024 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2"] Oct 30 13:44:34.105291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.105255 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:34.107920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.107891 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-k2hr2"] Oct 30 13:44:34.181512 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.181482 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66"] Oct 30 13:44:34.189451 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.189417 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-rhp66"] Oct 30 13:44:34.379929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.379891 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" podUID="b0bbdb7e-bee7-437c-b800-309359cfd827" containerName="payload-processing" containerID="cri-o://bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388" gracePeriod=25 Oct 30 13:44:34.419517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.419473 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" podUID="7e4b52fc-cf26-4470-ba92-2f1d6628d313" containerName="payload-pre-processing" containerID="cri-o://2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7" gracePeriod=25 Oct 30 13:44:34.451960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.451916 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48"] Oct 30 13:44:34.452432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.452397 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" podUID="2140b530-29d4-4965-8467-dbcabafe9b24" containerName="maas-api" containerID="cri-o://7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c" gracePeriod=30 Oct 30 13:44:34.589365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.589316 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:34.728505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.728481 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:44:34.768215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.768184 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:34.800742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.800709 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-metrics-certs\") pod \"2140b530-29d4-4965-8467-dbcabafe9b24\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " Oct 30 13:44:34.800915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.800897 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2140b530-29d4-4965-8467-dbcabafe9b24-kube-api-access-xm95q\" (UniqueName: \"kubernetes.io/projected/2140b530-29d4-4965-8467-dbcabafe9b24-kube-api-access-xm95q\") pod \"2140b530-29d4-4965-8467-dbcabafe9b24\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " Oct 30 13:44:34.800975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.800940 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-maas-api-tls\") pod \"2140b530-29d4-4965-8467-dbcabafe9b24\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " Oct 30 13:44:34.801068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.801051 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2140b530-29d4-4965-8467-dbcabafe9b24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2140b530-29d4-4965-8467-dbcabafe9b24-openshift-service-ca\") pod \"2140b530-29d4-4965-8467-dbcabafe9b24\" (UID: \"2140b530-29d4-4965-8467-dbcabafe9b24\") " Oct 30 13:44:34.802010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.801981 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2140b530-29d4-4965-8467-dbcabafe9b24-openshift-service-ca" pod "2140b530-29d4-4965-8467-dbcabafe9b24" (UID: "2140b530-29d4-4965-8467-dbcabafe9b24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:34.803822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.803798 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2140b530-29d4-4965-8467-dbcabafe9b24-kube-api-access-xm95q" pod "2140b530-29d4-4965-8467-dbcabafe9b24" (UID: "2140b530-29d4-4965-8467-dbcabafe9b24"). InnerVolumeSpecName "kube-api-access-xm95q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:34.804149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.804119 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-maas-api-tls" pod "2140b530-29d4-4965-8467-dbcabafe9b24" (UID: "2140b530-29d4-4965-8467-dbcabafe9b24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:34.804252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.804200 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-metrics-certs" pod "2140b530-29d4-4965-8467-dbcabafe9b24" (UID: "2140b530-29d4-4965-8467-dbcabafe9b24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:34.813444 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.813361 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv"] Oct 30 13:44:34.822410 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.822387 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9c6033-2wsjv"] Oct 30 13:44:34.904372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.902622 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:34.904372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.902659 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm95q\" (UniqueName: \"kubernetes.io/projected/2140b530-29d4-4965-8467-dbcabafe9b24-kube-api-access-xm95q\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:34.904372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.902675 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140b530-29d4-4965-8467-dbcabafe9b24-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:34.904372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.902693 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2140b530-29d4-4965-8467-dbcabafe9b24-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:34.943814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:34.943771 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:35.027790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.027707 5049 generic.go:356] "Generic (PLEG): container finished" podID="2140b530-29d4-4965-8467-dbcabafe9b24" containerID="7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c" exitCode=0 Oct 30 13:44:35.027790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.027777 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" Oct 30 13:44:35.028272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.027785 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" event={"ID":"2140b530-29d4-4965-8467-dbcabafe9b24","Type":"ContainerDied","Data":"7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c"} Oct 30 13:44:35.028272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.027924 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48" event={"ID":"2140b530-29d4-4965-8467-dbcabafe9b24","Type":"ContainerDied","Data":"258d561b077059547055ec4c3f354292f41895c782d06b6fecd2bbc9ec682808"} Oct 30 13:44:35.028272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.027961 5049 scope.go:122] "RemoveContainer" containerID="7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c" Oct 30 13:44:35.041457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.041428 5049 scope.go:122] "RemoveContainer" containerID="7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c" Oct 30 13:44:35.041795 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:35.041772 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c\": container with ID starting with 7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c not found: ID does not exist" containerID="7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c" Oct 30 13:44:35.041874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.041806 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c"} err="failed to get container status \"7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c\": rpc error: code = NotFound desc = could not find container \"7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c\": container with ID starting with 7127de07288ac7b2a584946d684966085ef62c3d3162a39b81a83c025649d60c not found: ID does not exist" Oct 30 13:44:35.084854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.084823 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48"] Oct 30 13:44:35.093357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.093315 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82796d-845d757c8f-kzs48"] Oct 30 13:44:35.190716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.190687 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:35.241392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.241360 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:35.243495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.243470 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:35.308612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.308576 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-xks-ca\") pod \"41593460-aec9-43dd-a4ad-27be23c7b607\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " Oct 30 13:44:35.308813 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.308681 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-openshift-ca\") pod \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " Oct 30 13:44:35.308967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.308824 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-xks-ca\") pod \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " Oct 30 13:44:35.308967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.308876 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41593460-aec9-43dd-a4ad-27be23c7b607-kube-api-access-j6s6z\" (UniqueName: \"kubernetes.io/projected/41593460-aec9-43dd-a4ad-27be23c7b607-kube-api-access-j6s6z\") pod \"41593460-aec9-43dd-a4ad-27be23c7b607\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " Oct 30 13:44:35.308967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.308939 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-kube-api-access-9wpjs\" (UniqueName: \"kubernetes.io/projected/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-kube-api-access-9wpjs\") pod \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\" (UID: \"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1\") " Oct 30 13:44:35.309139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.308984 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-openshift-ca\") pod \"41593460-aec9-43dd-a4ad-27be23c7b607\" (UID: \"41593460-aec9-43dd-a4ad-27be23c7b607\") " Oct 30 13:44:35.309139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.309105 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-openshift-ca" pod "65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1" (UID: "65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:35.309574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.309539 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-openshift-ca" pod "41593460-aec9-43dd-a4ad-27be23c7b607" (UID: "41593460-aec9-43dd-a4ad-27be23c7b607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:35.309682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.309583 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:35.311834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.311797 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-xks-ca" pod "41593460-aec9-43dd-a4ad-27be23c7b607" (UID: "41593460-aec9-43dd-a4ad-27be23c7b607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:35.311962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.311841 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41593460-aec9-43dd-a4ad-27be23c7b607-kube-api-access-j6s6z" pod "41593460-aec9-43dd-a4ad-27be23c7b607" (UID: "41593460-aec9-43dd-a4ad-27be23c7b607"). InnerVolumeSpecName "kube-api-access-j6s6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:35.312260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.312232 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-kube-api-access-9wpjs" pod "65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1" (UID: "65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1"). InnerVolumeSpecName "kube-api-access-9wpjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:35.312532 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.312514 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-xks-ca" pod "65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1" (UID: "65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:35.326822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.326781 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc"] Oct 30 13:44:35.350495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.350459 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2140b530-29d4-4965-8467-dbcabafe9b24" path="/var/lib/kubelet/pods/2140b530-29d4-4965-8467-dbcabafe9b24/volumes" Oct 30 13:44:35.350956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.350941 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715c24ba-f9e7-4389-ba41-59b7ea89e582" path="/var/lib/kubelet/pods/715c24ba-f9e7-4389-ba41-59b7ea89e582/volumes" Oct 30 13:44:35.351432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.351416 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" path="/var/lib/kubelet/pods/71949dbf-4b7a-45e7-bd4f-775596a65fbd/volumes" Oct 30 13:44:35.352291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.352269 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3ab600-2ed6-40aa-8bb4-cef6afbee938" path="/var/lib/kubelet/pods/9c3ab600-2ed6-40aa-8bb4-cef6afbee938/volumes" Oct 30 13:44:35.352729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.352697 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:35.353007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.352989 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82796d-lthsc"] Oct 30 13:44:35.410433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.410394 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wpjs\" (UniqueName: \"kubernetes.io/projected/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-kube-api-access-9wpjs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:35.410433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.410435 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:35.410604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.410452 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41593460-aec9-43dd-a4ad-27be23c7b607-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:35.410604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.410467 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:35.410604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.410480 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6s6z\" (UniqueName: \"kubernetes.io/projected/41593460-aec9-43dd-a4ad-27be23c7b607-kube-api-access-j6s6z\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:35.508146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.508110 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:35.861378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:35.861340 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:36.020174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.020142 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2"] Oct 30 13:44:36.040052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.040016 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" Oct 30 13:44:36.040500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.040049 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs" event={"ID":"41593460-aec9-43dd-a4ad-27be23c7b607","Type":"ContainerDied","Data":"16310b184de8dab6db2b98c9fb3bbbca0e2e15f516c1765ef1dd3db0829463a7"} Oct 30 13:44:36.040500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.040107 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16310b184de8dab6db2b98c9fb3bbbca0e2e15f516c1765ef1dd3db0829463a7" Oct 30 13:44:36.043461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.043435 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" Oct 30 13:44:36.043600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.043455 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd" event={"ID":"65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1","Type":"ContainerDied","Data":"2a5339f613e2c8d6c557e0db3117c510382e9dde6768c2880fcb763e83212726"} Oct 30 13:44:36.043600 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.043500 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a5339f613e2c8d6c557e0db3117c510382e9dde6768c2880fcb763e83212726" Oct 30 13:44:36.055435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.055408 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48"] Oct 30 13:44:36.310044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.309991 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:36.559740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.559707 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:36.732268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:36.731497 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" podUID="561552f7-0505-4fda-9fd1-eb1fa882a262" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:37.164373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:37.164303 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:37.337674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:37.337640 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9250b2b2-0fda-4ac9-afdd-1efc4819bba2" path="/var/lib/kubelet/pods/9250b2b2-0fda-4ac9-afdd-1efc4819bba2/volumes" Oct 30 13:44:37.741514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:37.741475 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" podUID="14634916-26ab-4c10-bfba-03002f7a4eab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:37.846155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:37.846126 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:37.914021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:37.913979 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb"] Oct 30 13:44:37.924145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:37.924114 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6da60a-97mnb"] Oct 30 13:44:38.002643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:38.002541 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:38.048019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:38.047983 5049 scope.go:122] "RemoveContainer" containerID="549dc5fa949b1354973ab53be109d93b7278a87488a1fbf2f952ad12033a52f5" Oct 30 13:44:38.058802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:38.058782 5049 scope.go:122] "RemoveContainer" containerID="f9747a572cf44477dc5217d1d45f5c89bc78c65b0207a83d8bec8e4ebb920748" Oct 30 13:44:38.068708 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:38.068689 5049 scope.go:122] "RemoveContainer" containerID="7bf7443e7b81189fdfe17e74f183e29abe5c1bd8a033cab39608f420ffc8a74c" Oct 30 13:44:38.427440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:38.427391 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:38.722223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:38.722134 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp"] Oct 30 13:44:38.903199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:38.903162 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:39.058593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:39.058550 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:39.344265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:39.344172 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3974584-4132-47aa-81c4-a988d90789d4" path="/var/lib/kubelet/pods/f3974584-4132-47aa-81c4-a988d90789d4/volumes" Oct 30 13:44:39.744373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:39.744259 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:40.363957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:40.363900 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:40.471615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:40.471579 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4"] Oct 30 13:44:40.480770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:40.480733 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-8d82d5-zmdk4"] Oct 30 13:44:40.542181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:40.542148 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:40.815342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:40.815285 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" podUID="7e4b52fc-cf26-4470-ba92-2f1d6628d313" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:40.815806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:40.815401 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" podUID="b0bbdb7e-bee7-437c-b800-309359cfd827" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:41.003378 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.003343 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:41.020914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.020877 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" podUID="d05e65bd-2ee8-4b2e-9391-aa04ae888377" containerName="payload-processing" containerID="cri-o://83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635" gracePeriod=25 Oct 30 13:44:41.056178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.056145 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" podUID="f671b2e1-1907-476a-b1e8-6509e55ca6f6" containerName="payload-pre-processing" containerID="cri-o://139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13" gracePeriod=25 Oct 30 13:44:41.340606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.340571 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46911daa-69f0-4084-92c6-1a555be94f61" path="/var/lib/kubelet/pods/46911daa-69f0-4084-92c6-1a555be94f61/volumes" Oct 30 13:44:41.340802 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.340777 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:44:41.389816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.389790 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:44:41.479210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.479175 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f671b2e1-1907-476a-b1e8-6509e55ca6f6-kube-api-access-snm8d\" (UniqueName: \"kubernetes.io/projected/f671b2e1-1907-476a-b1e8-6509e55ca6f6-kube-api-access-snm8d\") pod \"f671b2e1-1907-476a-b1e8-6509e55ca6f6\" (UID: \"f671b2e1-1907-476a-b1e8-6509e55ca6f6\") " Oct 30 13:44:41.479430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.479261 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f671b2e1-1907-476a-b1e8-6509e55ca6f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f671b2e1-1907-476a-b1e8-6509e55ca6f6-plugins-config-volume\") pod \"f671b2e1-1907-476a-b1e8-6509e55ca6f6\" (UID: \"f671b2e1-1907-476a-b1e8-6509e55ca6f6\") " Oct 30 13:44:41.479430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.479302 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05e65bd-2ee8-4b2e-9391-aa04ae888377-kube-api-access-bbhd8\" (UniqueName: \"kubernetes.io/projected/d05e65bd-2ee8-4b2e-9391-aa04ae888377-kube-api-access-bbhd8\") pod \"d05e65bd-2ee8-4b2e-9391-aa04ae888377\" (UID: \"d05e65bd-2ee8-4b2e-9391-aa04ae888377\") " Oct 30 13:44:41.479572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.479467 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05e65bd-2ee8-4b2e-9391-aa04ae888377-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05e65bd-2ee8-4b2e-9391-aa04ae888377-plugins-config-volume\") pod \"d05e65bd-2ee8-4b2e-9391-aa04ae888377\" (UID: \"d05e65bd-2ee8-4b2e-9391-aa04ae888377\") " Oct 30 13:44:41.479812 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.479700 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f671b2e1-1907-476a-b1e8-6509e55ca6f6-plugins-config-volume" pod "f671b2e1-1907-476a-b1e8-6509e55ca6f6" (UID: "f671b2e1-1907-476a-b1e8-6509e55ca6f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:41.479925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.479899 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05e65bd-2ee8-4b2e-9391-aa04ae888377-plugins-config-volume" pod "d05e65bd-2ee8-4b2e-9391-aa04ae888377" (UID: "d05e65bd-2ee8-4b2e-9391-aa04ae888377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:41.479995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.479908 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f671b2e1-1907-476a-b1e8-6509e55ca6f6-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:41.481593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.481565 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05e65bd-2ee8-4b2e-9391-aa04ae888377-kube-api-access-bbhd8" pod "d05e65bd-2ee8-4b2e-9391-aa04ae888377" (UID: "d05e65bd-2ee8-4b2e-9391-aa04ae888377"). InnerVolumeSpecName "kube-api-access-bbhd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:41.481593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.481578 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f671b2e1-1907-476a-b1e8-6509e55ca6f6-kube-api-access-snm8d" pod "f671b2e1-1907-476a-b1e8-6509e55ca6f6" (UID: "f671b2e1-1907-476a-b1e8-6509e55ca6f6"). InnerVolumeSpecName "kube-api-access-snm8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:41.580992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.580959 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbhd8\" (UniqueName: \"kubernetes.io/projected/d05e65bd-2ee8-4b2e-9391-aa04ae888377-kube-api-access-bbhd8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:41.580992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.580987 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05e65bd-2ee8-4b2e-9391-aa04ae888377-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:41.580992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:41.580996 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snm8d\" (UniqueName: \"kubernetes.io/projected/f671b2e1-1907-476a-b1e8-6509e55ca6f6-kube-api-access-snm8d\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:42.089055 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.089017 5049 generic.go:356] "Generic (PLEG): container finished" podID="f671b2e1-1907-476a-b1e8-6509e55ca6f6" containerID="139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13" exitCode=0 Oct 30 13:44:42.089534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.089068 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" Oct 30 13:44:42.089534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.089103 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" event={"ID":"f671b2e1-1907-476a-b1e8-6509e55ca6f6","Type":"ContainerDied","Data":"139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13"} Oct 30 13:44:42.089534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.089157 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48" event={"ID":"f671b2e1-1907-476a-b1e8-6509e55ca6f6","Type":"ContainerDied","Data":"9897b2d90f9f75d1ffd2de5c876ec3626766670584e6f74968e28f0be2a540cb"} Oct 30 13:44:42.089534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.089186 5049 scope.go:122] "RemoveContainer" containerID="139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13" Oct 30 13:44:42.090762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.090735 5049 generic.go:356] "Generic (PLEG): container finished" podID="d05e65bd-2ee8-4b2e-9391-aa04ae888377" containerID="83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635" exitCode=0 Oct 30 13:44:42.090862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.090771 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" Oct 30 13:44:42.090862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.090819 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" event={"ID":"d05e65bd-2ee8-4b2e-9391-aa04ae888377","Type":"ContainerDied","Data":"83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635"} Oct 30 13:44:42.090958 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.090868 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2" event={"ID":"d05e65bd-2ee8-4b2e-9391-aa04ae888377","Type":"ContainerDied","Data":"be3d580e01ade1ed7d47b6c816073026232c06912f8ce9c7f91096d3ff30f82b"} Oct 30 13:44:42.105865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.105842 5049 scope.go:122] "RemoveContainer" containerID="139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13" Oct 30 13:44:42.106203 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:42.106180 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13\": container with ID starting with 139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13 not found: ID does not exist" containerID="139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13" Oct 30 13:44:42.106265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.106213 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13"} err="failed to get container status \"139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13\": rpc error: code = NotFound desc = could not find container \"139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13\": container with ID starting with 139cb60aeb7b1037cc5760ce85c63f05e48b2aef679b9b4556ab534b4278bf13 not found: ID does not exist" Oct 30 13:44:42.106265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.106232 5049 scope.go:122] "RemoveContainer" containerID="83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635" Oct 30 13:44:42.119435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.119416 5049 scope.go:122] "RemoveContainer" containerID="83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635" Oct 30 13:44:42.119748 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:42.119723 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635\": container with ID starting with 83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635 not found: ID does not exist" containerID="83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635" Oct 30 13:44:42.119832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.119760 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635"} err="failed to get container status \"83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635\": rpc error: code = NotFound desc = could not find container \"83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635\": container with ID starting with 83d28b3295c45b9d8bc5371470aba7a769df4f972520ee682935f79c049cd635 not found: ID does not exist" Oct 30 13:44:42.155606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.155579 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2"] Oct 30 13:44:42.161015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.160993 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6da60a-5754789b94-2kqs2"] Oct 30 13:44:42.191421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.191393 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48"] Oct 30 13:44:42.194835 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.194815 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6da60a-5449f9f7c5-v2z48"] Oct 30 13:44:42.563167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:42.563127 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:43.146132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:43.146090 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:43.339352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:43.339284 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05e65bd-2ee8-4b2e-9391-aa04ae888377" path="/var/lib/kubelet/pods/d05e65bd-2ee8-4b2e-9391-aa04ae888377/volumes" Oct 30 13:44:43.339912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:43.339891 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f671b2e1-1907-476a-b1e8-6509e55ca6f6" path="/var/lib/kubelet/pods/f671b2e1-1907-476a-b1e8-6509e55ca6f6/volumes" Oct 30 13:44:43.349367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:43.349334 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:43.532402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:43.532295 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:43.723145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:43.723063 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" podUID="e483618c-00a8-458a-b8e8-bc927c60ba52" containerName="payload-processing" containerID="cri-o://348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47" gracePeriod=25 Oct 30 13:44:43.886996 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:43.886968 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.236590 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq"] Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237413 5049 cpu_manager.go:414] "containerMap: removing container" podUID="41593460-aec9-43dd-a4ad-27be23c7b607" containerName="revoke-keys" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237430 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41593460-aec9-43dd-a4ad-27be23c7b607" containerName="revoke-keys" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237452 5049 cpu_manager.go:414] "containerMap: removing container" podUID="715c24ba-f9e7-4389-ba41-59b7ea89e582" containerName="istio-proxy" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237462 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715c24ba-f9e7-4389-ba41-59b7ea89e582" containerName="istio-proxy" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237490 5049 cpu_manager.go:414] "containerMap: removing container" podUID="2140b530-29d4-4965-8467-dbcabafe9b24" containerName="maas-api" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237498 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2140b530-29d4-4965-8467-dbcabafe9b24" containerName="maas-api" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237528 5049 cpu_manager.go:414] "containerMap: removing container" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237537 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237550 5049 cpu_manager.go:414] "containerMap: removing container" podUID="46911daa-69f0-4084-92c6-1a555be94f61" containerName="revoke-keys" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237558 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46911daa-69f0-4084-92c6-1a555be94f61" containerName="revoke-keys" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237571 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f671b2e1-1907-476a-b1e8-6509e55ca6f6" containerName="payload-pre-processing" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237579 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f671b2e1-1907-476a-b1e8-6509e55ca6f6" containerName="payload-pre-processing" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237603 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d05e65bd-2ee8-4b2e-9391-aa04ae888377" containerName="payload-processing" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237610 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05e65bd-2ee8-4b2e-9391-aa04ae888377" containerName="payload-processing" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237627 5049 cpu_manager.go:414] "containerMap: removing container" podUID="65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1" containerName="revoke-keys" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237635 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1" containerName="revoke-keys" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237744 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="2140b530-29d4-4965-8467-dbcabafe9b24" containerName="maas-api" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237761 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="71949dbf-4b7a-45e7-bd4f-775596a65fbd" containerName="istio-proxy" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237772 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="46911daa-69f0-4084-92c6-1a555be94f61" containerName="revoke-keys" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237782 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f671b2e1-1907-476a-b1e8-6509e55ca6f6" containerName="payload-pre-processing" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237796 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05e65bd-2ee8-4b2e-9391-aa04ae888377" containerName="payload-processing" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237807 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1" containerName="revoke-keys" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237819 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="41593460-aec9-43dd-a4ad-27be23c7b607" containerName="revoke-keys" Oct 30 13:44:44.241433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.237827 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="715c24ba-f9e7-4389-ba41-59b7ea89e582" containerName="istio-proxy" Oct 30 13:44:44.261805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.261767 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq"] Oct 30 13:44:44.262003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.261885 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.271636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.271598 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:44.285635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.285609 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:44:44.285785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.285659 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:44:44.336251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.335977 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkvhm\" (UniqueName: \"kubernetes.io/projected/7e503802-8d23-4b8c-a357-34c464db3be3-kube-api-access-nkvhm\") pod \"maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.336251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.336041 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.336251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.336121 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.437128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.437079 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkvhm\" (UniqueName: \"kubernetes.io/projected/7e503802-8d23-4b8c-a357-34c464db3be3-kube-api-access-nkvhm\") pod \"maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.437331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.437148 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.437331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.437194 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.437871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.437843 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.439721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.439689 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.465783 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.465753 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkvhm\" (UniqueName: \"kubernetes.io/projected/7e503802-8d23-4b8c-a357-34c464db3be3-kube-api-access-nkvhm\") pod \"maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.539576 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.539488 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8"] Oct 30 13:44:44.543930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.543910 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.567984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.567954 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8"] Oct 30 13:44:44.578246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.578216 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:44.641454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.639942 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.641454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.640060 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.641454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.640131 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv9cw\" (UniqueName: \"kubernetes.io/projected/84de9a2c-5d62-4ab2-a922-360fdffec6ee-kube-api-access-kv9cw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.741508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.741471 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.741666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.741536 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9cw\" (UniqueName: \"kubernetes.io/projected/84de9a2c-5d62-4ab2-a922-360fdffec6ee-kube-api-access-kv9cw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.741740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.741671 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.742212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.742190 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.744247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.744222 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.770706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.770679 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9cw\" (UniqueName: \"kubernetes.io/projected/84de9a2c-5d62-4ab2-a922-360fdffec6ee-kube-api-access-kv9cw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.847908 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.847864 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:44.861889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.861859 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:44.903362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:44.903316 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq"] Oct 30 13:44:44.907257 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:44:44.907214 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e503802_8d23_4b8c_a357_34c464db3be3.slice/crio-7e71722125d9db790889fddded7c0b2d6a16bb2016c57b4e498c44c986447b42 WatchSource:0}: Error finding container 7e71722125d9db790889fddded7c0b2d6a16bb2016c57b4e498c44c986447b42: Status 404 returned error can't find the container with id 7e71722125d9db790889fddded7c0b2d6a16bb2016c57b4e498c44c986447b42 Oct 30 13:44:45.118648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:45.118560 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" event={"ID":"7e503802-8d23-4b8c-a357-34c464db3be3","Type":"ContainerStarted","Data":"7e71722125d9db790889fddded7c0b2d6a16bb2016c57b4e498c44c986447b42"} Oct 30 13:44:45.180888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:45.180846 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8"] Oct 30 13:44:45.184242 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:44:45.184205 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84de9a2c_5d62_4ab2_a922_360fdffec6ee.slice/crio-7dedc4956c58fef3b56d506868d16193995db5abc33909a69ecca2ad7367cd15 WatchSource:0}: Error finding container 7dedc4956c58fef3b56d506868d16193995db5abc33909a69ecca2ad7367cd15: Status 404 returned error can't find the container with id 7dedc4956c58fef3b56d506868d16193995db5abc33909a69ecca2ad7367cd15 Oct 30 13:44:45.998391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:45.998302 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:44:46.058671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.058632 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14634916-26ab-4c10-bfba-03002f7a4eab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14634916-26ab-4c10-bfba-03002f7a4eab-plugins-config-volume\") pod \"14634916-26ab-4c10-bfba-03002f7a4eab\" (UID: \"14634916-26ab-4c10-bfba-03002f7a4eab\") " Oct 30 13:44:46.058887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.058701 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14634916-26ab-4c10-bfba-03002f7a4eab-kube-api-access-wzwsh\" (UniqueName: \"kubernetes.io/projected/14634916-26ab-4c10-bfba-03002f7a4eab-kube-api-access-wzwsh\") pod \"14634916-26ab-4c10-bfba-03002f7a4eab\" (UID: \"14634916-26ab-4c10-bfba-03002f7a4eab\") " Oct 30 13:44:46.059062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.059037 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14634916-26ab-4c10-bfba-03002f7a4eab-plugins-config-volume" pod "14634916-26ab-4c10-bfba-03002f7a4eab" (UID: "14634916-26ab-4c10-bfba-03002f7a4eab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:46.059263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.059238 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14634916-26ab-4c10-bfba-03002f7a4eab-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:46.065042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.065017 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14634916-26ab-4c10-bfba-03002f7a4eab-kube-api-access-wzwsh" pod "14634916-26ab-4c10-bfba-03002f7a4eab" (UID: "14634916-26ab-4c10-bfba-03002f7a4eab"). InnerVolumeSpecName "kube-api-access-wzwsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:46.106924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.106891 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:44:46.134403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.134368 5049 generic.go:356] "Generic (PLEG): container finished" podID="14634916-26ab-4c10-bfba-03002f7a4eab" containerID="9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c" exitCode=0 Oct 30 13:44:46.134403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.134408 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" Oct 30 13:44:46.134646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.134445 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" event={"ID":"14634916-26ab-4c10-bfba-03002f7a4eab","Type":"ContainerDied","Data":"9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c"} Oct 30 13:44:46.134646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.134503 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr" event={"ID":"14634916-26ab-4c10-bfba-03002f7a4eab","Type":"ContainerDied","Data":"fe46cb852ba694562d56322c904530ec00a1fdbc0e878adf795fcd814ad9d153"} Oct 30 13:44:46.134646 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.134532 5049 scope.go:122] "RemoveContainer" containerID="9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c" Oct 30 13:44:46.136278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.136255 5049 generic.go:356] "Generic (PLEG): container finished" podID="7e503802-8d23-4b8c-a357-34c464db3be3" containerID="16bc54f506752a7eefa44ebc5a4b2fc8c886252e388a09dad60a8ef9cd84ebcf" exitCode=0 Oct 30 13:44:46.136430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.136359 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" event={"ID":"7e503802-8d23-4b8c-a357-34c464db3be3","Type":"ContainerDied","Data":"16bc54f506752a7eefa44ebc5a4b2fc8c886252e388a09dad60a8ef9cd84ebcf"} Oct 30 13:44:46.140091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.138094 5049 generic.go:356] "Generic (PLEG): container finished" podID="561552f7-0505-4fda-9fd1-eb1fa882a262" containerID="526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4" exitCode=0 Oct 30 13:44:46.140091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.138290 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" Oct 30 13:44:46.140091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.138464 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" event={"ID":"561552f7-0505-4fda-9fd1-eb1fa882a262","Type":"ContainerDied","Data":"526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4"} Oct 30 13:44:46.140091 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.138502 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk" event={"ID":"561552f7-0505-4fda-9fd1-eb1fa882a262","Type":"ContainerDied","Data":"07a66aa4ada2ddb89bdcb79a628bd9327b3f4b09912c03d89ac10e9e534abef2"} Oct 30 13:44:46.145699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.145670 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" event={"ID":"84de9a2c-5d62-4ab2-a922-360fdffec6ee","Type":"ContainerStarted","Data":"f971778356a646a3cf7cd39242ed6a2e90d2d38f24ae539cb9dab94dc9654871"} Oct 30 13:44:46.145797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.145712 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" event={"ID":"84de9a2c-5d62-4ab2-a922-360fdffec6ee","Type":"ContainerStarted","Data":"7dedc4956c58fef3b56d506868d16193995db5abc33909a69ecca2ad7367cd15"} Oct 30 13:44:46.149066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.149039 5049 scope.go:122] "RemoveContainer" containerID="9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c" Oct 30 13:44:46.149381 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:46.149355 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c\": container with ID starting with 9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c not found: ID does not exist" containerID="9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c" Oct 30 13:44:46.149491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.149392 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c"} err="failed to get container status \"9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c\": rpc error: code = NotFound desc = could not find container \"9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c\": container with ID starting with 9492eff6a208d1b02ff6fd2edb028bd41057ec1f7fcec5f580810d7ce0e3387c not found: ID does not exist" Oct 30 13:44:46.149491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.149416 5049 scope.go:122] "RemoveContainer" containerID="526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4" Oct 30 13:44:46.159757 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.159729 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561552f7-0505-4fda-9fd1-eb1fa882a262-kube-api-access-df5c2\" (UniqueName: \"kubernetes.io/projected/561552f7-0505-4fda-9fd1-eb1fa882a262-kube-api-access-df5c2\") pod \"561552f7-0505-4fda-9fd1-eb1fa882a262\" (UID: \"561552f7-0505-4fda-9fd1-eb1fa882a262\") " Oct 30 13:44:46.159931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.159909 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561552f7-0505-4fda-9fd1-eb1fa882a262-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561552f7-0505-4fda-9fd1-eb1fa882a262-plugins-config-volume\") pod \"561552f7-0505-4fda-9fd1-eb1fa882a262\" (UID: \"561552f7-0505-4fda-9fd1-eb1fa882a262\") " Oct 30 13:44:46.160514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.160490 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzwsh\" (UniqueName: \"kubernetes.io/projected/14634916-26ab-4c10-bfba-03002f7a4eab-kube-api-access-wzwsh\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:46.161040 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.161025 5049 scope.go:122] "RemoveContainer" containerID="526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4" Oct 30 13:44:46.161357 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:46.161306 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4\": container with ID starting with 526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4 not found: ID does not exist" containerID="526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4" Oct 30 13:44:46.161474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.161370 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4"} err="failed to get container status \"526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4\": rpc error: code = NotFound desc = could not find container \"526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4\": container with ID starting with 526a9fd1ee1bbd6ed0b475933cf105d18f49ece73130251b53ea4c5f26d384b4 not found: ID does not exist" Oct 30 13:44:46.161551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.161531 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561552f7-0505-4fda-9fd1-eb1fa882a262-plugins-config-volume" pod "561552f7-0505-4fda-9fd1-eb1fa882a262" (UID: "561552f7-0505-4fda-9fd1-eb1fa882a262"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:46.162907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.162880 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561552f7-0505-4fda-9fd1-eb1fa882a262-kube-api-access-df5c2" pod "561552f7-0505-4fda-9fd1-eb1fa882a262" (UID: "561552f7-0505-4fda-9fd1-eb1fa882a262"). InnerVolumeSpecName "kube-api-access-df5c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:46.256440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.256360 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr"] Oct 30 13:44:46.261616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.261586 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561552f7-0505-4fda-9fd1-eb1fa882a262-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:46.261616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.261615 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df5c2\" (UniqueName: \"kubernetes.io/projected/561552f7-0505-4fda-9fd1-eb1fa882a262-kube-api-access-df5c2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:46.262016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.261990 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d833bd-54ddcdf987-bm2sr"] Oct 30 13:44:46.302397 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.302313 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" podStartSLOduration=2.302294599 podStartE2EDuration="2.302294599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:46.299489486 +0000 UTC m=+3193.114683909" watchObservedRunningTime="2026-10-30 13:44:46.302294599 +0000 UTC m=+3193.117489019" Oct 30 13:44:46.493164 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.493124 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk"] Oct 30 13:44:46.495674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.495646 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:46.498541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:46.498515 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d833bd-cb8d9565b-d8ntk"] Oct 30 13:44:47.158419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.158385 5049 generic.go:356] "Generic (PLEG): container finished" podID="84de9a2c-5d62-4ab2-a922-360fdffec6ee" containerID="f971778356a646a3cf7cd39242ed6a2e90d2d38f24ae539cb9dab94dc9654871" exitCode=0 Oct 30 13:44:47.158906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.158476 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" event={"ID":"84de9a2c-5d62-4ab2-a922-360fdffec6ee","Type":"ContainerDied","Data":"f971778356a646a3cf7cd39242ed6a2e90d2d38f24ae539cb9dab94dc9654871"} Oct 30 13:44:47.212942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.212883 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:47.334146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.334122 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:47.337982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.337953 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14634916-26ab-4c10-bfba-03002f7a4eab" path="/var/lib/kubelet/pods/14634916-26ab-4c10-bfba-03002f7a4eab/volumes" Oct 30 13:44:47.338845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.338819 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="561552f7-0505-4fda-9fd1-eb1fa882a262" path="/var/lib/kubelet/pods/561552f7-0505-4fda-9fd1-eb1fa882a262/volumes" Oct 30 13:44:47.372568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.372533 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc"] Oct 30 13:44:47.376762 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.376734 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-xks-ca\") pod \"7e503802-8d23-4b8c-a357-34c464db3be3\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " Oct 30 13:44:47.376916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.376868 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-openshift-ca\") pod \"7e503802-8d23-4b8c-a357-34c464db3be3\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " Oct 30 13:44:47.376968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.376956 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e503802-8d23-4b8c-a357-34c464db3be3-kube-api-access-nkvhm\" (UniqueName: \"kubernetes.io/projected/7e503802-8d23-4b8c-a357-34c464db3be3-kube-api-access-nkvhm\") pod \"7e503802-8d23-4b8c-a357-34c464db3be3\" (UID: \"7e503802-8d23-4b8c-a357-34c464db3be3\") " Oct 30 13:44:47.377212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.377185 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-openshift-ca" pod "7e503802-8d23-4b8c-a357-34c464db3be3" (UID: "7e503802-8d23-4b8c-a357-34c464db3be3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:47.377346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.377292 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:47.379567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.379543 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-xks-ca" pod "7e503802-8d23-4b8c-a357-34c464db3be3" (UID: "7e503802-8d23-4b8c-a357-34c464db3be3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:47.379567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.379556 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e503802-8d23-4b8c-a357-34c464db3be3-kube-api-access-nkvhm" pod "7e503802-8d23-4b8c-a357-34c464db3be3" (UID: "7e503802-8d23-4b8c-a357-34c464db3be3"). InnerVolumeSpecName "kube-api-access-nkvhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:47.383978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.383954 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d833bd-wb2zc"] Oct 30 13:44:47.469382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.469269 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:47.478749 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.478717 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkvhm\" (UniqueName: \"kubernetes.io/projected/7e503802-8d23-4b8c-a357-34c464db3be3-kube-api-access-nkvhm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:47.478749 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.478748 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e503802-8d23-4b8c-a357-34c464db3be3-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:47.671985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.671953 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:47.921468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:47.921431 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:48.089019 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.088986 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:48.169254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.169223 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" Oct 30 13:44:48.169745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.169261 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq" event={"ID":"7e503802-8d23-4b8c-a357-34c464db3be3","Type":"ContainerDied","Data":"7e71722125d9db790889fddded7c0b2d6a16bb2016c57b4e498c44c986447b42"} Oct 30 13:44:48.169745 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.169306 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e71722125d9db790889fddded7c0b2d6a16bb2016c57b4e498c44c986447b42" Oct 30 13:44:48.343678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.343652 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" podUID="e483618c-00a8-458a-b8e8-bc927c60ba52" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:48.367888 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.367859 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:48.388700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.388671 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-openshift-ca\") pod \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " Oct 30 13:44:48.388841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.388708 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-xks-ca\") pod \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " Oct 30 13:44:48.388893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.388865 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84de9a2c-5d62-4ab2-a922-360fdffec6ee-kube-api-access-kv9cw\" (UniqueName: \"kubernetes.io/projected/84de9a2c-5d62-4ab2-a922-360fdffec6ee-kube-api-access-kv9cw\") pod \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\" (UID: \"84de9a2c-5d62-4ab2-a922-360fdffec6ee\") " Oct 30 13:44:48.389074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.389054 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-openshift-ca" pod "84de9a2c-5d62-4ab2-a922-360fdffec6ee" (UID: "84de9a2c-5d62-4ab2-a922-360fdffec6ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:48.389570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.389545 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.391038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.391014 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-xks-ca" pod "84de9a2c-5d62-4ab2-a922-360fdffec6ee" (UID: "84de9a2c-5d62-4ab2-a922-360fdffec6ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:48.391038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.391023 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84de9a2c-5d62-4ab2-a922-360fdffec6ee-kube-api-access-kv9cw" pod "84de9a2c-5d62-4ab2-a922-360fdffec6ee" (UID: "84de9a2c-5d62-4ab2-a922-360fdffec6ee"). InnerVolumeSpecName "kube-api-access-kv9cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:48.491077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.491004 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv9cw\" (UniqueName: \"kubernetes.io/projected/84de9a2c-5d62-4ab2-a922-360fdffec6ee-kube-api-access-kv9cw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.491077 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.491032 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84de9a2c-5d62-4ab2-a922-360fdffec6ee-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.915165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:48.915128 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s"] Oct 30 13:44:49.095948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:49.095908 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:49.177689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:49.177606 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" Oct 30 13:44:49.177689 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:49.177647 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8" event={"ID":"84de9a2c-5d62-4ab2-a922-360fdffec6ee","Type":"ContainerDied","Data":"7dedc4956c58fef3b56d506868d16193995db5abc33909a69ecca2ad7367cd15"} Oct 30 13:44:49.178212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:49.177698 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dedc4956c58fef3b56d506868d16193995db5abc33909a69ecca2ad7367cd15" Oct 30 13:44:49.249073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:49.249043 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:49.336505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:49.336468 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac2db76-8e24-4899-aa7c-71755b76ead6" path="/var/lib/kubelet/pods/2ac2db76-8e24-4899-aa7c-71755b76ead6/volumes" Oct 30 13:44:49.490154 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:49.490060 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt"] Oct 30 13:44:49.490359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:49.490257 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" containerID="cri-o://d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d" gracePeriod=30 Oct 30 13:44:49.574334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:49.574300 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:49.986589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:49.986550 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:50.118268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118235 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp"] Oct 30 13:44:50.118808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118793 5049 cpu_manager.go:414] "containerMap: removing container" podUID="84de9a2c-5d62-4ab2-a922-360fdffec6ee" containerName="revoke-keys" Oct 30 13:44:50.118856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118810 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84de9a2c-5d62-4ab2-a922-360fdffec6ee" containerName="revoke-keys" Oct 30 13:44:50.118856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118830 5049 cpu_manager.go:414] "containerMap: removing container" podUID="7e503802-8d23-4b8c-a357-34c464db3be3" containerName="revoke-keys" Oct 30 13:44:50.118856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118835 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e503802-8d23-4b8c-a357-34c464db3be3" containerName="revoke-keys" Oct 30 13:44:50.118856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118849 5049 cpu_manager.go:414] "containerMap: removing container" podUID="561552f7-0505-4fda-9fd1-eb1fa882a262" containerName="payload-processing" Oct 30 13:44:50.118856 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118854 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561552f7-0505-4fda-9fd1-eb1fa882a262" containerName="payload-processing" Oct 30 13:44:50.119043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118866 5049 cpu_manager.go:414] "containerMap: removing container" podUID="14634916-26ab-4c10-bfba-03002f7a4eab" containerName="payload-pre-processing" Oct 30 13:44:50.119043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118871 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14634916-26ab-4c10-bfba-03002f7a4eab" containerName="payload-pre-processing" Oct 30 13:44:50.119043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118958 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="84de9a2c-5d62-4ab2-a922-360fdffec6ee" containerName="revoke-keys" Oct 30 13:44:50.119043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118970 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e503802-8d23-4b8c-a357-34c464db3be3" containerName="revoke-keys" Oct 30 13:44:50.119043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118978 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="561552f7-0505-4fda-9fd1-eb1fa882a262" containerName="payload-processing" Oct 30 13:44:50.119043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.118984 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="14634916-26ab-4c10-bfba-03002f7a4eab" containerName="payload-pre-processing" Oct 30 13:44:50.122579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.122553 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.135572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.135539 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:44:50.146058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.146030 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:44:50.149431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.149403 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:50.149850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.149829 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp"] Oct 30 13:44:50.210931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.210898 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rx4k\" (UniqueName: \"kubernetes.io/projected/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-kube-api-access-7rx4k\") pod \"maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.211489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.210979 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.211489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.211176 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.312464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.312431 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.312658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.312564 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.312658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.312612 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rx4k\" (UniqueName: \"kubernetes.io/projected/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-kube-api-access-7rx4k\") pod \"maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.313294 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.313264 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.315045 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.315017 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.334119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.334083 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rx4k\" (UniqueName: \"kubernetes.io/projected/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-kube-api-access-7rx4k\") pod \"maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.437159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.437116 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:50.758032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.757946 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp"] Oct 30 13:44:50.761986 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:44:50.761958 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a09da6_0d7c_4c9c_b11e_1ee66eaed1a1.slice/crio-35d8e0c784e516c7ad38788506998d56e4c9e5bba852c1d268cce094097da957 WatchSource:0}: Error finding container 35d8e0c784e516c7ad38788506998d56e4c9e5bba852c1d268cce094097da957: Status 404 returned error can't find the container with id 35d8e0c784e516c7ad38788506998d56e4c9e5bba852c1d268cce094097da957 Oct 30 13:44:50.815026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.814995 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" podUID="7e4b52fc-cf26-4470-ba92-2f1d6628d313" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:50.815151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:50.815020 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" podUID="b0bbdb7e-bee7-437c-b800-309359cfd827" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:51.199521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:51.198463 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" event={"ID":"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1","Type":"ContainerStarted","Data":"b4b934459abbed5c3e017f51e41a64934352d57587f8d36a1f77fca221ac2340"} Oct 30 13:44:51.199521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:51.198531 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" event={"ID":"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1","Type":"ContainerStarted","Data":"35d8e0c784e516c7ad38788506998d56e4c9e5bba852c1d268cce094097da957"} Oct 30 13:44:51.288815 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:51.288736 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:51.500796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:51.500764 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:52.219499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:52.219463 5049 generic.go:356] "Generic (PLEG): container finished" podID="40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1" containerID="b4b934459abbed5c3e017f51e41a64934352d57587f8d36a1f77fca221ac2340" exitCode=0 Oct 30 13:44:52.219706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:52.219545 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" event={"ID":"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1","Type":"ContainerDied","Data":"b4b934459abbed5c3e017f51e41a64934352d57587f8d36a1f77fca221ac2340"} Oct 30 13:44:52.306053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:52.306007 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:53.119305 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.119261 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn"] Oct 30 13:44:53.130313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.130282 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r"] Oct 30 13:44:53.287151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.287121 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:53.387748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.387681 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:53.461754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.461718 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-openshift-ca\") pod \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " Oct 30 13:44:53.461935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.461766 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-kube-api-access-7rx4k\" (UniqueName: \"kubernetes.io/projected/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-kube-api-access-7rx4k\") pod \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " Oct 30 13:44:53.461935 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.461861 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-xks-ca\") pod \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\" (UID: \"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1\") " Oct 30 13:44:53.462211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.462182 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-openshift-ca" pod "40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1" (UID: "40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:53.462743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.462577 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:53.464211 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.464190 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-xks-ca" pod "40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1" (UID: "40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:53.464554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.464533 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-kube-api-access-7rx4k" pod "40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1" (UID: "40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1"). InnerVolumeSpecName "kube-api-access-7rx4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:53.563459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.563426 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rx4k\" (UniqueName: \"kubernetes.io/projected/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-kube-api-access-7rx4k\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:53.563459 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:53.563458 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.237352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.237312 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" Oct 30 13:44:54.237517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.237352 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp" event={"ID":"40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1","Type":"ContainerDied","Data":"35d8e0c784e516c7ad38788506998d56e4c9e5bba852c1d268cce094097da957"} Oct 30 13:44:54.237517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.237400 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35d8e0c784e516c7ad38788506998d56e4c9e5bba852c1d268cce094097da957" Oct 30 13:44:54.352339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.352293 5049 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:44:54.352557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.352386 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:44:54.762389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.751562 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2"] Oct 30 13:44:54.762389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.759503 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:44:54.880753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.880715 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bbdb7e-bee7-437c-b800-309359cfd827-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bbdb7e-bee7-437c-b800-309359cfd827-plugins-config-volume\") pod \"b0bbdb7e-bee7-437c-b800-309359cfd827\" (UID: \"b0bbdb7e-bee7-437c-b800-309359cfd827\") " Oct 30 13:44:54.880956 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.880797 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bbdb7e-bee7-437c-b800-309359cfd827-kube-api-access-7dgvk\" (UniqueName: \"kubernetes.io/projected/b0bbdb7e-bee7-437c-b800-309359cfd827-kube-api-access-7dgvk\") pod \"b0bbdb7e-bee7-437c-b800-309359cfd827\" (UID: \"b0bbdb7e-bee7-437c-b800-309359cfd827\") " Oct 30 13:44:54.887503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.881383 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bbdb7e-bee7-437c-b800-309359cfd827-plugins-config-volume" pod "b0bbdb7e-bee7-437c-b800-309359cfd827" (UID: "b0bbdb7e-bee7-437c-b800-309359cfd827"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:54.887503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.884820 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bbdb7e-bee7-437c-b800-309359cfd827-kube-api-access-7dgvk" pod "b0bbdb7e-bee7-437c-b800-309359cfd827" (UID: "b0bbdb7e-bee7-437c-b800-309359cfd827"). InnerVolumeSpecName "kube-api-access-7dgvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:54.894026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.893990 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:54.900704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.900291 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:44:54.905004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.904924 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:44:54.981970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.981935 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-credential-socket\") pod \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " Oct 30 13:44:54.982132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982028 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istiod-ca-cert\") pod \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " Oct 30 13:44:54.982132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982060 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-data\") pod \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " Oct 30 13:44:54.982132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982087 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4b52fc-cf26-4470-ba92-2f1d6628d313-kube-api-access-zcmqn\" (UniqueName: \"kubernetes.io/projected/7e4b52fc-cf26-4470-ba92-2f1d6628d313-kube-api-access-zcmqn\") pod \"7e4b52fc-cf26-4470-ba92-2f1d6628d313\" (UID: \"7e4b52fc-cf26-4470-ba92-2f1d6628d313\") " Oct 30 13:44:54.982132 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982129 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-podinfo\") pod \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " Oct 30 13:44:54.982559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982538 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-kube-api-access-j87cf\" (UniqueName: \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-kube-api-access-j87cf\") pod \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " Oct 30 13:44:54.982607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982580 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4b52fc-cf26-4470-ba92-2f1d6628d313-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4b52fc-cf26-4470-ba92-2f1d6628d313-plugins-config-volume\") pod \"7e4b52fc-cf26-4470-ba92-2f1d6628d313\" (UID: \"7e4b52fc-cf26-4470-ba92-2f1d6628d313\") " Oct 30 13:44:54.982871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982610 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-token\" (UniqueName: \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-token\") pod \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " Oct 30 13:44:54.982871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982633 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-socket\") pod \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " Oct 30 13:44:54.982871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982661 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-envoy\") pod \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " Oct 30 13:44:54.982871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982725 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-certs\") pod \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\" (UID: \"852ca5dd-65ad-4ee2-a3b4-b38203b81454\") " Oct 30 13:44:54.983082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.982884 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-credential-socket" pod "852ca5dd-65ad-4ee2-a3b4-b38203b81454" (UID: "852ca5dd-65ad-4ee2-a3b4-b38203b81454"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:54.983119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.983097 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-certs" pod "852ca5dd-65ad-4ee2-a3b4-b38203b81454" (UID: "852ca5dd-65ad-4ee2-a3b4-b38203b81454"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:54.983236 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.983221 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.983278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.983237 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.983278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.983247 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bbdb7e-bee7-437c-b800-309359cfd827-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.983391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.983280 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dgvk\" (UniqueName: \"kubernetes.io/projected/b0bbdb7e-bee7-437c-b800-309359cfd827-kube-api-access-7dgvk\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.983470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.983428 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-socket" pod "852ca5dd-65ad-4ee2-a3b4-b38203b81454" (UID: "852ca5dd-65ad-4ee2-a3b4-b38203b81454"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:54.984155 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.984126 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4b52fc-cf26-4470-ba92-2f1d6628d313-plugins-config-volume" pod "7e4b52fc-cf26-4470-ba92-2f1d6628d313" (UID: "7e4b52fc-cf26-4470-ba92-2f1d6628d313"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:54.985931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.985902 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4b52fc-cf26-4470-ba92-2f1d6628d313-kube-api-access-zcmqn" pod "7e4b52fc-cf26-4470-ba92-2f1d6628d313" (UID: "7e4b52fc-cf26-4470-ba92-2f1d6628d313"). InnerVolumeSpecName "kube-api-access-zcmqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:54.986044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.986011 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-envoy" pod "852ca5dd-65ad-4ee2-a3b4-b38203b81454" (UID: "852ca5dd-65ad-4ee2-a3b4-b38203b81454"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:54.986597 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.986561 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-podinfo" pod "852ca5dd-65ad-4ee2-a3b4-b38203b81454" (UID: "852ca5dd-65ad-4ee2-a3b4-b38203b81454"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:54.986781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.986760 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-kube-api-access-j87cf" pod "852ca5dd-65ad-4ee2-a3b4-b38203b81454" (UID: "852ca5dd-65ad-4ee2-a3b4-b38203b81454"). InnerVolumeSpecName "kube-api-access-j87cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:54.987214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.987185 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-token" pod "852ca5dd-65ad-4ee2-a3b4-b38203b81454" (UID: "852ca5dd-65ad-4ee2-a3b4-b38203b81454"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:54.989136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.989114 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istiod-ca-cert" pod "852ca5dd-65ad-4ee2-a3b4-b38203b81454" (UID: "852ca5dd-65ad-4ee2-a3b4-b38203b81454"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:54.989341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.989302 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m"] Oct 30 13:44:54.989430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:54.989365 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-data" pod "852ca5dd-65ad-4ee2-a3b4-b38203b81454" (UID: "852ca5dd-65ad-4ee2-a3b4-b38203b81454"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:55.076946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.076913 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:55.084333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.084278 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j87cf\" (UniqueName: \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-kube-api-access-j87cf\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.084333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.084332 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4b52fc-cf26-4470-ba92-2f1d6628d313-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.084333 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.084347 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.084549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.084359 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.084549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.084367 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.084549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.084376 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.084549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.084384 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.084549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.084392 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcmqn\" (UniqueName: \"kubernetes.io/projected/7e4b52fc-cf26-4470-ba92-2f1d6628d313-kube-api-access-zcmqn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.084549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.084400 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852ca5dd-65ad-4ee2-a3b4-b38203b81454-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.245896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.245855 5049 generic.go:356] "Generic (PLEG): container finished" podID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerID="d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d" exitCode=0 Oct 30 13:44:55.246106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.245901 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" Oct 30 13:44:55.246106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.245947 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" event={"ID":"852ca5dd-65ad-4ee2-a3b4-b38203b81454","Type":"ContainerDied","Data":"d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d"} Oct 30 13:44:55.246106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.246007 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt" event={"ID":"852ca5dd-65ad-4ee2-a3b4-b38203b81454","Type":"ContainerDied","Data":"445c86a1fd1880c46ced2eaba4ceed75cda362b2e6b454c95243dd2931d45346"} Oct 30 13:44:55.246106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.246037 5049 scope.go:122] "RemoveContainer" containerID="d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d" Oct 30 13:44:55.247543 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.247522 5049 generic.go:356] "Generic (PLEG): container finished" podID="7e4b52fc-cf26-4470-ba92-2f1d6628d313" containerID="2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7" exitCode=0 Oct 30 13:44:55.247636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.247568 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" Oct 30 13:44:55.247694 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.247571 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" event={"ID":"7e4b52fc-cf26-4470-ba92-2f1d6628d313","Type":"ContainerDied","Data":"2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7"} Oct 30 13:44:55.247748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.247691 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v" event={"ID":"7e4b52fc-cf26-4470-ba92-2f1d6628d313","Type":"ContainerDied","Data":"086a539349cd254d9b369cb81751620e2f9b02d12cdb0ad8a177adb2f3d5f306"} Oct 30 13:44:55.249273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.249251 5049 generic.go:356] "Generic (PLEG): container finished" podID="b0bbdb7e-bee7-437c-b800-309359cfd827" containerID="bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388" exitCode=0 Oct 30 13:44:55.249437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.249287 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" Oct 30 13:44:55.249437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.249348 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" event={"ID":"b0bbdb7e-bee7-437c-b800-309359cfd827","Type":"ContainerDied","Data":"bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388"} Oct 30 13:44:55.249437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.249381 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk" event={"ID":"b0bbdb7e-bee7-437c-b800-309359cfd827","Type":"ContainerDied","Data":"5f641afcc2a4ed3b55e2efad9f5ef69d7ff8d2c2481ef8e129f9de79db659f70"} Oct 30 13:44:55.259365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.259299 5049 scope.go:122] "RemoveContainer" containerID="d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d" Oct 30 13:44:55.259662 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:55.259627 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d\": container with ID starting with d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d not found: ID does not exist" containerID="d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d" Oct 30 13:44:55.259723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.259676 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d"} err="failed to get container status \"d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d\": rpc error: code = NotFound desc = could not find container \"d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d\": container with ID starting with d716e649f42e7e01ea3015c5ac35728b840a42fadf10a73ac60fe4b7dfd7150d not found: ID does not exist" Oct 30 13:44:55.259723 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.259701 5049 scope.go:122] "RemoveContainer" containerID="2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7" Oct 30 13:44:55.272400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.272379 5049 scope.go:122] "RemoveContainer" containerID="2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7" Oct 30 13:44:55.272847 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:55.272813 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7\": container with ID starting with 2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7 not found: ID does not exist" containerID="2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7" Oct 30 13:44:55.272950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.272851 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7"} err="failed to get container status \"2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7\": rpc error: code = NotFound desc = could not find container \"2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7\": container with ID starting with 2b734f694313a3685b1959e939964fb45bfb89cf927f9cdc36e245111bc121b7 not found: ID does not exist" Oct 30 13:44:55.272950 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.272871 5049 scope.go:122] "RemoveContainer" containerID="bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388" Oct 30 13:44:55.283076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.283057 5049 scope.go:122] "RemoveContainer" containerID="bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388" Oct 30 13:44:55.283351 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:55.283315 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388\": container with ID starting with bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388 not found: ID does not exist" containerID="bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388" Oct 30 13:44:55.283400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.283359 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388"} err="failed to get container status \"bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388\": rpc error: code = NotFound desc = could not find container \"bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388\": container with ID starting with bee743cf46dbe1012378816549ce641f1b07ed3c362710ee4198dbccaf753388 not found: ID does not exist" Oct 30 13:44:55.313808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.313769 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk"] Oct 30 13:44:55.327542 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.326031 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82796d-5544c659bd-mqmkk"] Oct 30 13:44:55.345544 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.345513 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bbdb7e-bee7-437c-b800-309359cfd827" path="/var/lib/kubelet/pods/b0bbdb7e-bee7-437c-b800-309359cfd827/volumes" Oct 30 13:44:55.349189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.349165 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:55.359938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.359912 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v"] Oct 30 13:44:55.364481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.364457 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82796d-6cc6dcc975-8jl2v"] Oct 30 13:44:55.401131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.401103 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt"] Oct 30 13:44:55.407237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.407208 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-pmrzt"] Oct 30 13:44:55.626205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.626116 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v"] Oct 30 13:44:55.665117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.665083 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj"] Oct 30 13:44:55.820925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.820889 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:55.996238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:55.996146 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:56.198677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:56.198643 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:56.677988 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:44:56.677949 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:44:57.088379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:57.088315 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:57.312049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:57.312016 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:57.335810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:57.335779 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4b52fc-cf26-4470-ba92-2f1d6628d313" path="/var/lib/kubelet/pods/7e4b52fc-cf26-4470-ba92-2f1d6628d313/volumes" Oct 30 13:44:57.336247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:57.336234 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" path="/var/lib/kubelet/pods/852ca5dd-65ad-4ee2-a3b4-b38203b81454/volumes" Oct 30 13:44:57.720625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:57.720591 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:57.748395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:57.748364 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md"] Oct 30 13:44:57.748589 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:57.748566 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" containerID="cri-o://62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95" gracePeriod=30 Oct 30 13:44:57.905743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:57.905696 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:58.343982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.343946 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" podUID="e483618c-00a8-458a-b8e8-bc927c60ba52" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:58.565555 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.565521 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2"] Oct 30 13:44:58.606275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.606188 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj"] Oct 30 13:44:58.758718 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.758680 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:58.865186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.865112 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs"] Oct 30 13:44:58.865913 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.865896 5049 cpu_manager.go:414] "containerMap: removing container" podUID="40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1" containerName="revoke-keys" Oct 30 13:44:58.865967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.865917 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1" containerName="revoke-keys" Oct 30 13:44:58.865967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.865941 5049 cpu_manager.go:414] "containerMap: removing container" podUID="b0bbdb7e-bee7-437c-b800-309359cfd827" containerName="payload-processing" Oct 30 13:44:58.865967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.865949 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bbdb7e-bee7-437c-b800-309359cfd827" containerName="payload-processing" Oct 30 13:44:58.866114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.865985 5049 cpu_manager.go:414] "containerMap: removing container" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" Oct 30 13:44:58.866114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.865993 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" Oct 30 13:44:58.866114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.866015 5049 cpu_manager.go:414] "containerMap: removing container" podUID="7e4b52fc-cf26-4470-ba92-2f1d6628d313" containerName="payload-pre-processing" Oct 30 13:44:58.866114 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.866023 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4b52fc-cf26-4470-ba92-2f1d6628d313" containerName="payload-pre-processing" Oct 30 13:44:58.866265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.866163 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4b52fc-cf26-4470-ba92-2f1d6628d313" containerName="payload-pre-processing" Oct 30 13:44:58.866265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.866182 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1" containerName="revoke-keys" Oct 30 13:44:58.866265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.866192 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bbdb7e-bee7-437c-b800-309359cfd827" containerName="payload-processing" Oct 30 13:44:58.866265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.866206 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="852ca5dd-65ad-4ee2-a3b4-b38203b81454" containerName="istio-proxy" Oct 30 13:44:58.871780 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.871752 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:58.899131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.893999 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:44:58.899131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.894042 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:44:58.900004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.899976 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs"] Oct 30 13:44:58.924648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.924611 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9n9q\" (UniqueName: \"kubernetes.io/projected/937fd81f-4191-49ed-9a59-216ea01544c5-kube-api-access-j9n9q\") pod \"maas-api-revoke-sub-e168c9e7-pcwgs\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:58.924805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.924665 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e168c9e7-pcwgs\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:58.924873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.924794 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e168c9e7-pcwgs\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:58.941223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:58.941195 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:59.026536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:59.026499 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9n9q\" (UniqueName: \"kubernetes.io/projected/937fd81f-4191-49ed-9a59-216ea01544c5-kube-api-access-j9n9q\") pod \"maas-api-revoke-sub-e168c9e7-pcwgs\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:59.026719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:59.026569 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e168c9e7-pcwgs\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:59.026719 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:59.026614 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e168c9e7-pcwgs\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:59.027284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:59.027255 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e168c9e7-pcwgs\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:59.029113 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:59.029092 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e168c9e7-pcwgs\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:59.047855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:59.047830 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9n9q\" (UniqueName: \"kubernetes.io/projected/937fd81f-4191-49ed-9a59-216ea01544c5-kube-api-access-j9n9q\") pod \"maas-api-revoke-sub-e168c9e7-pcwgs\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:59.186455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:59.186362 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:44:59.211036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:59.210989 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:59.508157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:59.508018 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs"] Oct 30 13:44:59.513064 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:44:59.513033 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937fd81f_4191_49ed_9a59_216ea01544c5.slice/crio-49d066f60f888a229ac50e40989bab9640b74a845cbfa75a525e56567c53143a WatchSource:0}: Error finding container 49d066f60f888a229ac50e40989bab9640b74a845cbfa75a525e56567c53143a: Status 404 returned error can't find the container with id 49d066f60f888a229ac50e40989bab9640b74a845cbfa75a525e56567c53143a Oct 30 13:44:59.752245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:44:59.752162 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" podUID="3fa7700f-de5f-4302-8147-edb31d44f888" containerName="payload-processing" containerID="cri-o://e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6" gracePeriod=25 Oct 30 13:45:00.158331 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.158215 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6"] Oct 30 13:45:00.164424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.163310 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" Oct 30 13:45:00.168367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.164847 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl"] Oct 30 13:45:00.170897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.170869 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" Oct 30 13:45:00.201307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.201275 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6"] Oct 30 13:45:00.210966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.210931 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl"] Oct 30 13:45:00.216523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.216493 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln"] Oct 30 13:45:00.222000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.221939 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" Oct 30 13:45:00.235096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.235062 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln"] Oct 30 13:45:00.241172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.241143 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6\" (UID: \"46fda2f4-2ef1-46c6-a85b-cfce89fe474e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" Oct 30 13:45:00.241332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.241177 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhwq\" (UniqueName: \"kubernetes.io/projected/c311b4f0-1429-4b35-ad65-250c5404b0c8-kube-api-access-vlhwq\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl\" (UID: \"c311b4f0-1429-4b35-ad65-250c5404b0c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" Oct 30 13:45:00.241332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.241292 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szcbr\" (UniqueName: \"kubernetes.io/projected/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-kube-api-access-szcbr\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6\" (UID: \"46fda2f4-2ef1-46c6-a85b-cfce89fe474e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" Oct 30 13:45:00.241478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.241424 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c311b4f0-1429-4b35-ad65-250c5404b0c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl\" (UID: \"c311b4f0-1429-4b35-ad65-250c5404b0c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" Oct 30 13:45:00.297264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.297223 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" event={"ID":"937fd81f-4191-49ed-9a59-216ea01544c5","Type":"ContainerStarted","Data":"5d39c23654e2dddbad7bbfb2ad8c56eea45637399862187904af572dc16d5194"} Oct 30 13:45:00.297461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.297297 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" event={"ID":"937fd81f-4191-49ed-9a59-216ea01544c5","Type":"ContainerStarted","Data":"49d066f60f888a229ac50e40989bab9640b74a845cbfa75a525e56567c53143a"} Oct 30 13:45:00.342560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.342525 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9382dbcf-317a-47db-b952-59244215f86c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln\" (UID: \"9382dbcf-317a-47db-b952-59244215f86c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" Oct 30 13:45:00.342700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.342586 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhdbp\" (UniqueName: \"kubernetes.io/projected/9382dbcf-317a-47db-b952-59244215f86c-kube-api-access-rhdbp\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln\" (UID: \"9382dbcf-317a-47db-b952-59244215f86c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" Oct 30 13:45:00.342790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.342764 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6\" (UID: \"46fda2f4-2ef1-46c6-a85b-cfce89fe474e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" Oct 30 13:45:00.342857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.342814 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhwq\" (UniqueName: \"kubernetes.io/projected/c311b4f0-1429-4b35-ad65-250c5404b0c8-kube-api-access-vlhwq\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl\" (UID: \"c311b4f0-1429-4b35-ad65-250c5404b0c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" Oct 30 13:45:00.342924 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.342901 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szcbr\" (UniqueName: \"kubernetes.io/projected/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-kube-api-access-szcbr\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6\" (UID: \"46fda2f4-2ef1-46c6-a85b-cfce89fe474e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" Oct 30 13:45:00.343014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.342999 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c311b4f0-1429-4b35-ad65-250c5404b0c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl\" (UID: \"c311b4f0-1429-4b35-ad65-250c5404b0c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" Oct 30 13:45:00.343850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.343829 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6\" (UID: \"46fda2f4-2ef1-46c6-a85b-cfce89fe474e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" Oct 30 13:45:00.343952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.343930 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c311b4f0-1429-4b35-ad65-250c5404b0c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl\" (UID: \"c311b4f0-1429-4b35-ad65-250c5404b0c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" Oct 30 13:45:00.346015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.345970 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" podStartSLOduration=2.345954404 podStartE2EDuration="2.345954404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:00.342669809 +0000 UTC m=+3207.157864232" watchObservedRunningTime="2026-10-30 13:45:00.345954404 +0000 UTC m=+3207.161148840" Oct 30 13:45:00.367087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.367053 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhwq\" (UniqueName: \"kubernetes.io/projected/c311b4f0-1429-4b35-ad65-250c5404b0c8-kube-api-access-vlhwq\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl\" (UID: \"c311b4f0-1429-4b35-ad65-250c5404b0c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" Oct 30 13:45:00.367873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.367849 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szcbr\" (UniqueName: \"kubernetes.io/projected/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-kube-api-access-szcbr\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6\" (UID: \"46fda2f4-2ef1-46c6-a85b-cfce89fe474e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" Oct 30 13:45:00.444103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.444009 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdbp\" (UniqueName: \"kubernetes.io/projected/9382dbcf-317a-47db-b952-59244215f86c-kube-api-access-rhdbp\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln\" (UID: \"9382dbcf-317a-47db-b952-59244215f86c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" Oct 30 13:45:00.444288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.444157 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9382dbcf-317a-47db-b952-59244215f86c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln\" (UID: \"9382dbcf-317a-47db-b952-59244215f86c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" Oct 30 13:45:00.444805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.444776 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9382dbcf-317a-47db-b952-59244215f86c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln\" (UID: \"9382dbcf-317a-47db-b952-59244215f86c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" Oct 30 13:45:00.466127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.466092 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdbp\" (UniqueName: \"kubernetes.io/projected/9382dbcf-317a-47db-b952-59244215f86c-kube-api-access-rhdbp\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln\" (UID: \"9382dbcf-317a-47db-b952-59244215f86c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" Oct 30 13:45:00.492529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.492496 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" Oct 30 13:45:00.505265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.505233 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" Oct 30 13:45:00.542163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.542128 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" Oct 30 13:45:00.543999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.543966 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm"] Oct 30 13:45:00.630554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.630514 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" podUID="91e5f412-4939-44cd-937d-2bf38707178e" containerName="payload-processing" containerID="cri-o://21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b" gracePeriod=25 Oct 30 13:45:00.665611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.665549 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" podUID="55f9f0d5-52a1-434f-acb9-fdbfe8671335" containerName="payload-pre-processing" containerID="cri-o://2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4" gracePeriod=25 Oct 30 13:45:00.740899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.740803 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:00.806460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.806426 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6"] Oct 30 13:45:00.810895 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:45:00.810854 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fda2f4_2ef1_46c6_a85b_cfce89fe474e.slice/crio-8c09dfdbb4a18d5b50ab2039b4692677c475c1229c2864bc86389f31b6eaa08a WatchSource:0}: Error finding container 8c09dfdbb4a18d5b50ab2039b4692677c475c1229c2864bc86389f31b6eaa08a: Status 404 returned error can't find the container with id 8c09dfdbb4a18d5b50ab2039b4692677c475c1229c2864bc86389f31b6eaa08a Oct 30 13:45:00.831884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.831856 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl"] Oct 30 13:45:00.836396 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:45:00.836362 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc311b4f0_1429_4b35_ad65_250c5404b0c8.slice/crio-031f53290cf5f869d459c957be986aa94b36bc9aec58100c98f75129a6c8197e WatchSource:0}: Error finding container 031f53290cf5f869d459c957be986aa94b36bc9aec58100c98f75129a6c8197e: Status 404 returned error can't find the container with id 031f53290cf5f869d459c957be986aa94b36bc9aec58100c98f75129a6c8197e Oct 30 13:45:00.904161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.904117 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln"] Oct 30 13:45:00.907704 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:45:00.907670 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9382dbcf_317a_47db_b952_59244215f86c.slice/crio-5b74c92e424c4de8e68b86368ca2632cdf650ecd7011c938c4ceae9847e548b7 WatchSource:0}: Error finding container 5b74c92e424c4de8e68b86368ca2632cdf650ecd7011c938c4ceae9847e548b7: Status 404 returned error can't find the container with id 5b74c92e424c4de8e68b86368ca2632cdf650ecd7011c938c4ceae9847e548b7 Oct 30 13:45:00.914999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:00.914975 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:01.117794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:01.117749 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:01.307285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:01.307249 5049 generic.go:356] "Generic (PLEG): container finished" podID="937fd81f-4191-49ed-9a59-216ea01544c5" containerID="5d39c23654e2dddbad7bbfb2ad8c56eea45637399862187904af572dc16d5194" exitCode=0 Oct 30 13:45:01.307999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:01.307364 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" event={"ID":"937fd81f-4191-49ed-9a59-216ea01544c5","Type":"ContainerDied","Data":"5d39c23654e2dddbad7bbfb2ad8c56eea45637399862187904af572dc16d5194"} Oct 30 13:45:01.308972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:01.308932 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" event={"ID":"9382dbcf-317a-47db-b952-59244215f86c","Type":"ContainerStarted","Data":"5b74c92e424c4de8e68b86368ca2632cdf650ecd7011c938c4ceae9847e548b7"} Oct 30 13:45:01.310873 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:01.310851 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" event={"ID":"c311b4f0-1429-4b35-ad65-250c5404b0c8","Type":"ContainerStarted","Data":"c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545"} Oct 30 13:45:01.310977 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:01.310895 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" event={"ID":"c311b4f0-1429-4b35-ad65-250c5404b0c8","Type":"ContainerStarted","Data":"031f53290cf5f869d459c957be986aa94b36bc9aec58100c98f75129a6c8197e"} Oct 30 13:45:01.312517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:01.312493 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" event={"ID":"46fda2f4-2ef1-46c6-a85b-cfce89fe474e","Type":"ContainerStarted","Data":"ab8606ee8c082b398234f91858886075de48b61365d47180f30315aa768ae585"} Oct 30 13:45:01.312607 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:01.312533 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" event={"ID":"46fda2f4-2ef1-46c6-a85b-cfce89fe474e","Type":"ContainerStarted","Data":"8c09dfdbb4a18d5b50ab2039b4692677c475c1229c2864bc86389f31b6eaa08a"} Oct 30 13:45:01.383547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:01.383478 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" podStartSLOduration=1.383458782 podStartE2EDuration="1.383458782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:01.382449569 +0000 UTC m=+3208.197643992" watchObservedRunningTime="2026-10-30 13:45:01.383458782 +0000 UTC m=+3208.198653207" Oct 30 13:45:01.422385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:01.422334 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" podStartSLOduration=1.422305363 podStartE2EDuration="1.422305363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:01.418493871 +0000 UTC m=+3208.233688295" watchObservedRunningTime="2026-10-30 13:45:01.422305363 +0000 UTC m=+3208.237499839" Oct 30 13:45:02.326692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.326658 5049 generic.go:356] "Generic (PLEG): container finished" podID="9382dbcf-317a-47db-b952-59244215f86c" containerID="7f515cd70253a26b0c4e3153adf48c95d5965c551108399128eb025e9f6a3fb8" exitCode=60 Oct 30 13:45:02.327157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.326749 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" event={"ID":"9382dbcf-317a-47db-b952-59244215f86c","Type":"ContainerDied","Data":"7f515cd70253a26b0c4e3153adf48c95d5965c551108399128eb025e9f6a3fb8"} Oct 30 13:45:02.327157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.327033 5049 scope.go:122] "RemoveContainer" containerID="7f515cd70253a26b0c4e3153adf48c95d5965c551108399128eb025e9f6a3fb8" Oct 30 13:45:02.328891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.328868 5049 generic.go:356] "Generic (PLEG): container finished" podID="c311b4f0-1429-4b35-ad65-250c5404b0c8" containerID="c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545" exitCode=60 Oct 30 13:45:02.328986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.328947 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" event={"ID":"c311b4f0-1429-4b35-ad65-250c5404b0c8","Type":"ContainerDied","Data":"c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545"} Oct 30 13:45:02.329194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.329176 5049 scope.go:122] "RemoveContainer" containerID="c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545" Oct 30 13:45:02.331003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.330981 5049 generic.go:356] "Generic (PLEG): container finished" podID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" containerID="ab8606ee8c082b398234f91858886075de48b61365d47180f30315aa768ae585" exitCode=60 Oct 30 13:45:02.331106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.331081 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" event={"ID":"46fda2f4-2ef1-46c6-a85b-cfce89fe474e","Type":"ContainerDied","Data":"ab8606ee8c082b398234f91858886075de48b61365d47180f30315aa768ae585"} Oct 30 13:45:02.331436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.331417 5049 scope.go:122] "RemoveContainer" containerID="ab8606ee8c082b398234f91858886075de48b61365d47180f30315aa768ae585" Oct 30 13:45:02.494027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.494003 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:45:02.567739 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.567699 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-xks-ca\") pod \"937fd81f-4191-49ed-9a59-216ea01544c5\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " Oct 30 13:45:02.567893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.567793 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-openshift-ca\") pod \"937fd81f-4191-49ed-9a59-216ea01544c5\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " Oct 30 13:45:02.567982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.567962 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937fd81f-4191-49ed-9a59-216ea01544c5-kube-api-access-j9n9q\" (UniqueName: \"kubernetes.io/projected/937fd81f-4191-49ed-9a59-216ea01544c5-kube-api-access-j9n9q\") pod \"937fd81f-4191-49ed-9a59-216ea01544c5\" (UID: \"937fd81f-4191-49ed-9a59-216ea01544c5\") " Oct 30 13:45:02.568166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.568144 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-openshift-ca" pod "937fd81f-4191-49ed-9a59-216ea01544c5" (UID: "937fd81f-4191-49ed-9a59-216ea01544c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:02.568392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.568368 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:02.569917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.569891 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-xks-ca" pod "937fd81f-4191-49ed-9a59-216ea01544c5" (UID: "937fd81f-4191-49ed-9a59-216ea01544c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:02.570112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.570085 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937fd81f-4191-49ed-9a59-216ea01544c5-kube-api-access-j9n9q" pod "937fd81f-4191-49ed-9a59-216ea01544c5" (UID: "937fd81f-4191-49ed-9a59-216ea01544c5"). InnerVolumeSpecName "kube-api-access-j9n9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:02.669805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.669770 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937fd81f-4191-49ed-9a59-216ea01544c5-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:02.669805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.669804 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9n9q\" (UniqueName: \"kubernetes.io/projected/937fd81f-4191-49ed-9a59-216ea01544c5-kube-api-access-j9n9q\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:02.674910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.674879 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw"] Oct 30 13:45:02.675110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.675087 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" podUID="20a120e6-66d4-4643-bd63-428f163eaf8c" containerName="maas-api" containerID="cri-o://f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d" gracePeriod=30 Oct 30 13:45:02.760414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.760380 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl"] Oct 30 13:45:02.837993 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:02.837961 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:03.020058 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.020031 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:45:03.080934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.080633 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-metrics-certs\") pod \"20a120e6-66d4-4643-bd63-428f163eaf8c\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " Oct 30 13:45:03.081866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.081838 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a120e6-66d4-4643-bd63-428f163eaf8c-kube-api-access-fr6b9\" (UniqueName: \"kubernetes.io/projected/20a120e6-66d4-4643-bd63-428f163eaf8c-kube-api-access-fr6b9\") pod \"20a120e6-66d4-4643-bd63-428f163eaf8c\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " Oct 30 13:45:03.082106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.082087 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-maas-api-tls\") pod \"20a120e6-66d4-4643-bd63-428f163eaf8c\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " Oct 30 13:45:03.085409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.083033 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a120e6-66d4-4643-bd63-428f163eaf8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a120e6-66d4-4643-bd63-428f163eaf8c-openshift-service-ca\") pod \"20a120e6-66d4-4643-bd63-428f163eaf8c\" (UID: \"20a120e6-66d4-4643-bd63-428f163eaf8c\") " Oct 30 13:45:03.085409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.083902 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a120e6-66d4-4643-bd63-428f163eaf8c-openshift-service-ca" pod "20a120e6-66d4-4643-bd63-428f163eaf8c" (UID: "20a120e6-66d4-4643-bd63-428f163eaf8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:03.086499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.086314 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a120e6-66d4-4643-bd63-428f163eaf8c-kube-api-access-fr6b9" pod "20a120e6-66d4-4643-bd63-428f163eaf8c" (UID: "20a120e6-66d4-4643-bd63-428f163eaf8c"). InnerVolumeSpecName "kube-api-access-fr6b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:03.086969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.086943 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-maas-api-tls" pod "20a120e6-66d4-4643-bd63-428f163eaf8c" (UID: "20a120e6-66d4-4643-bd63-428f163eaf8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:03.088230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.088206 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-metrics-certs" pod "20a120e6-66d4-4643-bd63-428f163eaf8c" (UID: "20a120e6-66d4-4643-bd63-428f163eaf8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:03.098740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.098720 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:45:03.182390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.182299 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx"] Oct 30 13:45:03.182920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.182897 5049 cpu_manager.go:414] "containerMap: removing container" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" Oct 30 13:45:03.182920 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.182917 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" Oct 30 13:45:03.183118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.182938 5049 cpu_manager.go:414] "containerMap: removing container" podUID="20a120e6-66d4-4643-bd63-428f163eaf8c" containerName="maas-api" Oct 30 13:45:03.183118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.182943 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a120e6-66d4-4643-bd63-428f163eaf8c" containerName="maas-api" Oct 30 13:45:03.183118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.182960 5049 cpu_manager.go:414] "containerMap: removing container" podUID="937fd81f-4191-49ed-9a59-216ea01544c5" containerName="revoke-keys" Oct 30 13:45:03.183118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.182966 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937fd81f-4191-49ed-9a59-216ea01544c5" containerName="revoke-keys" Oct 30 13:45:03.183118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.183062 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerName="istio-proxy" Oct 30 13:45:03.183118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.183075 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a120e6-66d4-4643-bd63-428f163eaf8c" containerName="maas-api" Oct 30 13:45:03.183118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.183082 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="937fd81f-4191-49ed-9a59-216ea01544c5" containerName="revoke-keys" Oct 30 13:45:03.184962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.184930 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-certs\") pod \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " Oct 30 13:45:03.185189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185004 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-socket\") pod \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " Oct 30 13:45:03.185189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185048 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-envoy\") pod \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " Oct 30 13:45:03.185189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185141 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-credential-socket\") pod \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " Oct 30 13:45:03.185429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185201 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-token\" (UniqueName: \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-token\") pod \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " Oct 30 13:45:03.185429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185266 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-data\") pod \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " Oct 30 13:45:03.185429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185350 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-kube-api-access-2qkws\" (UniqueName: \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-kube-api-access-2qkws\") pod \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " Oct 30 13:45:03.185429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185380 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istiod-ca-cert\") pod \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " Oct 30 13:45:03.185429 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185400 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-podinfo\") pod \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\" (UID: \"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610\") " Oct 30 13:45:03.185944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185922 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-maas-api-tls\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.186041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185950 5049 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a120e6-66d4-4643-bd63-428f163eaf8c-openshift-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.186041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185967 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a120e6-66d4-4643-bd63-428f163eaf8c-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.186041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.185980 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6b9\" (UniqueName: \"kubernetes.io/projected/20a120e6-66d4-4643-bd63-428f163eaf8c-kube-api-access-fr6b9\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.186951 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.186929 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.188941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.188762 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-certs" pod "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" (UID: "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:03.189098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.189076 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-socket" pod "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" (UID: "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:03.191103 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.190455 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-podinfo" pod "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" (UID: "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:03.191566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.191456 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-envoy" pod "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" (UID: "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:03.191566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.191519 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-data" pod "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" (UID: "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:03.191830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.191803 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-credential-socket" pod "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" (UID: "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:03.192001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.191980 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istiod-ca-cert" pod "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" (UID: "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:03.201725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.201693 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-kube-api-access-2qkws" pod "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" (UID: "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610"). InnerVolumeSpecName "kube-api-access-2qkws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:03.221686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.217087 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz"] Oct 30 13:45:03.222862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.222836 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx"] Oct 30 13:45:03.223135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.223115 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.243641 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.243595 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz"] Oct 30 13:45:03.247235 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.247189 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-token" pod "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" (UID: "a8fc97d9-1e7d-4b79-b70e-9036d3dfc610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:03.287188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287154 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50ee1507-fjszz\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.287346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287199 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50ee1507-fjszz\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.287409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287366 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac3bc12-jlntx\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.287409 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287398 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzwr\" (UniqueName: \"kubernetes.io/projected/7574c065-ecfa-4afc-b346-2519aa9be554-kube-api-access-wlzwr\") pod \"maas-api-revoke-sub-50ee1507-fjszz\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.287491 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287479 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac3bc12-jlntx\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.287551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287516 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28267\" (UniqueName: \"kubernetes.io/projected/46821723-84da-4e91-b3d1-e4de9f047b93-kube-api-access-28267\") pod \"maas-api-revoke-sub-4ac3bc12-jlntx\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.287611 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287598 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.287661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287613 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.287661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287633 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.287661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287643 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkws\" (UniqueName: \"kubernetes.io/projected/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-kube-api-access-2qkws\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.287661 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287658 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.287782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287666 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.287782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287675 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.287782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287684 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.287782 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.287691 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.288656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.288633 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:03.340521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.340490 5049 generic.go:356] "Generic (PLEG): container finished" podID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" containerID="62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95" exitCode=0 Oct 30 13:45:03.341001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.340547 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" Oct 30 13:45:03.341623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.340574 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" event={"ID":"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610","Type":"ContainerDied","Data":"62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95"} Oct 30 13:45:03.341751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.341662 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md" event={"ID":"a8fc97d9-1e7d-4b79-b70e-9036d3dfc610","Type":"ContainerDied","Data":"5657fe155bb935345525e3cf07da91593e4288caaf51f4d0de222bcd53b75f8a"} Oct 30 13:45:03.341751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.341700 5049 scope.go:122] "RemoveContainer" containerID="62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95" Oct 30 13:45:03.344250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.343793 5049 generic.go:356] "Generic (PLEG): container finished" podID="20a120e6-66d4-4643-bd63-428f163eaf8c" containerID="f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d" exitCode=0 Oct 30 13:45:03.344250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.343876 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" event={"ID":"20a120e6-66d4-4643-bd63-428f163eaf8c","Type":"ContainerDied","Data":"f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d"} Oct 30 13:45:03.344250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.343914 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" event={"ID":"20a120e6-66d4-4643-bd63-428f163eaf8c","Type":"ContainerDied","Data":"55340122a61f5898491913e8e16901625ca0820b63527df020e53105e7d55b88"} Oct 30 13:45:03.344250 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.344001 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw" Oct 30 13:45:03.353183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.353151 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" event={"ID":"937fd81f-4191-49ed-9a59-216ea01544c5","Type":"ContainerDied","Data":"49d066f60f888a229ac50e40989bab9640b74a845cbfa75a525e56567c53143a"} Oct 30 13:45:03.353315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.353263 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d066f60f888a229ac50e40989bab9640b74a845cbfa75a525e56567c53143a" Oct 30 13:45:03.353432 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.353417 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs" Oct 30 13:45:03.355737 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.355606 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" event={"ID":"9382dbcf-317a-47db-b952-59244215f86c","Type":"ContainerStarted","Data":"abfc09a5c9b6b740307f17e6c71fceebd8d63fbdbddc1303a2db367b36e662c8"} Oct 30 13:45:03.357830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.357805 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" podUID="c311b4f0-1429-4b35-ad65-250c5404b0c8" containerName="cleanup" containerID="cri-o://04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d" gracePeriod=30 Oct 30 13:45:03.358599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.358086 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" event={"ID":"c311b4f0-1429-4b35-ad65-250c5404b0c8","Type":"ContainerStarted","Data":"04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d"} Oct 30 13:45:03.360257 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.360234 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" event={"ID":"46fda2f4-2ef1-46c6-a85b-cfce89fe474e","Type":"ContainerStarted","Data":"ceb0da75b52a0da945cabf160bdbe39cd2a619cbe834e715a46eb6b91d5acf72"} Oct 30 13:45:03.380298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.380274 5049 scope.go:122] "RemoveContainer" containerID="62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95" Oct 30 13:45:03.380616 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:03.380594 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95\": container with ID starting with 62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95 not found: ID does not exist" containerID="62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95" Oct 30 13:45:03.380709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.380625 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95"} err="failed to get container status \"62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95\": rpc error: code = NotFound desc = could not find container \"62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95\": container with ID starting with 62a49c11b58120dad980197f230679ae521e3c993a5da8605d436563854e0a95 not found: ID does not exist" Oct 30 13:45:03.380709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.380645 5049 scope.go:122] "RemoveContainer" containerID="f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d" Oct 30 13:45:03.389021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.388992 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac3bc12-jlntx\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.389106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.389031 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzwr\" (UniqueName: \"kubernetes.io/projected/7574c065-ecfa-4afc-b346-2519aa9be554-kube-api-access-wlzwr\") pod \"maas-api-revoke-sub-50ee1507-fjszz\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.389106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.389063 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac3bc12-jlntx\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.389106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.389097 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28267\" (UniqueName: \"kubernetes.io/projected/46821723-84da-4e91-b3d1-e4de9f047b93-kube-api-access-28267\") pod \"maas-api-revoke-sub-4ac3bc12-jlntx\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.389219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.389147 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50ee1507-fjszz\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.389219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.389181 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50ee1507-fjszz\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.389669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.389651 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac3bc12-jlntx\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.390189 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.390167 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50ee1507-fjszz\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.391838 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.391812 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac3bc12-jlntx\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.391919 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.391896 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50ee1507-fjszz\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.393400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.393381 5049 scope.go:122] "RemoveContainer" containerID="f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d" Oct 30 13:45:03.393706 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:03.393684 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d\": container with ID starting with f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d not found: ID does not exist" containerID="f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d" Oct 30 13:45:03.393794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.393709 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d"} err="failed to get container status \"f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d\": rpc error: code = NotFound desc = could not find container \"f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d\": container with ID starting with f0b86f51aef2125f073cbb63d5678f9c3598012087ac8d05698b025f41778e3d not found: ID does not exist" Oct 30 13:45:03.420594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.420563 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28267\" (UniqueName: \"kubernetes.io/projected/46821723-84da-4e91-b3d1-e4de9f047b93-kube-api-access-28267\") pod \"maas-api-revoke-sub-4ac3bc12-jlntx\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.421671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.421637 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzwr\" (UniqueName: \"kubernetes.io/projected/7574c065-ecfa-4afc-b346-2519aa9be554-kube-api-access-wlzwr\") pod \"maas-api-revoke-sub-50ee1507-fjszz\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.446041 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.446016 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:03.509501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.509452 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" podStartSLOduration=3.509437766 podStartE2EDuration="3.509437766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.504912451 +0000 UTC m=+3210.320106875" watchObservedRunningTime="2026-10-30 13:45:03.509437766 +0000 UTC m=+3210.324632188" Oct 30 13:45:03.510047 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.510023 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:03.516863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.516306 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8"] Oct 30 13:45:03.522449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.522421 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-cda0d9-pb8r8"] Oct 30 13:45:03.541975 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.541944 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw"] Oct 30 13:45:03.550351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.548078 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-cda0d9-57bc7fddf5-gwkvw"] Oct 30 13:45:03.564285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.564260 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:03.566383 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.566300 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" podUID="ede810a9-3a02-490d-9a25-d0788bea4cdf" containerName="payload-processing" containerID="cri-o://344153ca7d7bd1782a71b1b1370fcd1ace0fe26860056380a2de00b4726f5113" gracePeriod=25 Oct 30 13:45:03.578538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.578508 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md"] Oct 30 13:45:03.586018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.585970 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-nr7md"] Oct 30 13:45:03.606980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.606937 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" podUID="409baa7e-092d-4d07-9a46-e611890a8458" containerName="payload-pre-processing" containerID="cri-o://2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050" gracePeriod=25 Oct 30 13:45:03.620865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.619538 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:03.797643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.797614 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:03.824685 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.824653 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx"] Oct 30 13:45:03.828589 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:45:03.828562 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46821723_84da_4e91_b3d1_e4de9f047b93.slice/crio-4b587220f0016a6bc6473f24986862092eb47437916725d9e84ae71aa52ed9ba WatchSource:0}: Error finding container 4b587220f0016a6bc6473f24986862092eb47437916725d9e84ae71aa52ed9ba: Status 404 returned error can't find the container with id 4b587220f0016a6bc6473f24986862092eb47437916725d9e84ae71aa52ed9ba Oct 30 13:45:03.920002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.919919 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz"] Oct 30 13:45:03.927281 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:45:03.927239 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7574c065_ecfa_4afc_b346_2519aa9be554.slice/crio-74b640c72cd0f8aef777be418237388f6f22fd80c46a6c39b7e31f1abd1d444e WatchSource:0}: Error finding container 74b640c72cd0f8aef777be418237388f6f22fd80c46a6c39b7e31f1abd1d444e: Status 404 returned error can't find the container with id 74b640c72cd0f8aef777be418237388f6f22fd80c46a6c39b7e31f1abd1d444e Oct 30 13:45:03.937301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:03.937154 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" containerID="cri-o://7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5" gracePeriod=45 Oct 30 13:45:04.076545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.076524 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:45:04.175362 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.175034 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:04.197841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.197804 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e483618c-00a8-458a-b8e8-bc927c60ba52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e483618c-00a8-458a-b8e8-bc927c60ba52-plugins-config-volume\") pod \"e483618c-00a8-458a-b8e8-bc927c60ba52\" (UID: \"e483618c-00a8-458a-b8e8-bc927c60ba52\") " Oct 30 13:45:04.198043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.197960 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e483618c-00a8-458a-b8e8-bc927c60ba52-kube-api-access-hvkbw\" (UniqueName: \"kubernetes.io/projected/e483618c-00a8-458a-b8e8-bc927c60ba52-kube-api-access-hvkbw\") pod \"e483618c-00a8-458a-b8e8-bc927c60ba52\" (UID: \"e483618c-00a8-458a-b8e8-bc927c60ba52\") " Oct 30 13:45:04.199222 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.199190 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e483618c-00a8-458a-b8e8-bc927c60ba52-plugins-config-volume" pod "e483618c-00a8-458a-b8e8-bc927c60ba52" (UID: "e483618c-00a8-458a-b8e8-bc927c60ba52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:04.200823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.200799 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e483618c-00a8-458a-b8e8-bc927c60ba52-kube-api-access-hvkbw" pod "e483618c-00a8-458a-b8e8-bc927c60ba52" (UID: "e483618c-00a8-458a-b8e8-bc927c60ba52"). InnerVolumeSpecName "kube-api-access-hvkbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:04.231300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.231276 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:45:04.298881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.298851 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-tmp-dir\") pod \"d191137c-94a6-4f18-b4f9-1c2e85de46de\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " Oct 30 13:45:04.299084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.298908 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-home\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-home\") pod \"d191137c-94a6-4f18-b4f9-1c2e85de46de\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " Oct 30 13:45:04.299084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.298964 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-dshm\") pod \"d191137c-94a6-4f18-b4f9-1c2e85de46de\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " Oct 30 13:45:04.299084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.299034 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d191137c-94a6-4f18-b4f9-1c2e85de46de-kube-api-access-mcmgr\" (UniqueName: \"kubernetes.io/projected/d191137c-94a6-4f18-b4f9-1c2e85de46de-kube-api-access-mcmgr\") pod \"d191137c-94a6-4f18-b4f9-1c2e85de46de\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " Oct 30 13:45:04.299262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.299135 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-tmp-dir" pod "d191137c-94a6-4f18-b4f9-1c2e85de46de" (UID: "d191137c-94a6-4f18-b4f9-1c2e85de46de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.299262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.299148 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-model-cache\") pod \"d191137c-94a6-4f18-b4f9-1c2e85de46de\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " Oct 30 13:45:04.299262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.299179 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d191137c-94a6-4f18-b4f9-1c2e85de46de-tls-certs\" (UniqueName: \"kubernetes.io/secret/d191137c-94a6-4f18-b4f9-1c2e85de46de-tls-certs\") pod \"d191137c-94a6-4f18-b4f9-1c2e85de46de\" (UID: \"d191137c-94a6-4f18-b4f9-1c2e85de46de\") " Oct 30 13:45:04.299608 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.299585 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvkbw\" (UniqueName: \"kubernetes.io/projected/e483618c-00a8-458a-b8e8-bc927c60ba52-kube-api-access-hvkbw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.299705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.299628 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e483618c-00a8-458a-b8e8-bc927c60ba52-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.299705 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.299644 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.302668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.301793 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-model-cache" pod "d191137c-94a6-4f18-b4f9-1c2e85de46de" (UID: "d191137c-94a6-4f18-b4f9-1c2e85de46de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.302668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.302056 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-home" pod "d191137c-94a6-4f18-b4f9-1c2e85de46de" (UID: "d191137c-94a6-4f18-b4f9-1c2e85de46de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.303184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.302289 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-dshm" pod "d191137c-94a6-4f18-b4f9-1c2e85de46de" (UID: "d191137c-94a6-4f18-b4f9-1c2e85de46de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.303277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.302316 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d191137c-94a6-4f18-b4f9-1c2e85de46de-kube-api-access-mcmgr" pod "d191137c-94a6-4f18-b4f9-1c2e85de46de" (UID: "d191137c-94a6-4f18-b4f9-1c2e85de46de"). InnerVolumeSpecName "kube-api-access-mcmgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:04.303566 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.303540 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d191137c-94a6-4f18-b4f9-1c2e85de46de-tls-certs" pod "d191137c-94a6-4f18-b4f9-1c2e85de46de" (UID: "d191137c-94a6-4f18-b4f9-1c2e85de46de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:04.374017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.373988 5049 generic.go:356] "Generic (PLEG): container finished" podID="9382dbcf-317a-47db-b952-59244215f86c" containerID="abfc09a5c9b6b740307f17e6c71fceebd8d63fbdbddc1303a2db367b36e662c8" exitCode=60 Oct 30 13:45:04.374416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.374255 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" event={"ID":"9382dbcf-317a-47db-b952-59244215f86c","Type":"ContainerDied","Data":"abfc09a5c9b6b740307f17e6c71fceebd8d63fbdbddc1303a2db367b36e662c8"} Oct 30 13:45:04.374416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.374316 5049 scope.go:122] "RemoveContainer" containerID="7f515cd70253a26b0c4e3153adf48c95d5965c551108399128eb025e9f6a3fb8" Oct 30 13:45:04.374513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.374448 5049 scope.go:122] "RemoveContainer" containerID="abfc09a5c9b6b740307f17e6c71fceebd8d63fbdbddc1303a2db367b36e662c8" Oct 30 13:45:04.374800 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:04.374768 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln_odh-ai-gateway-infra(9382dbcf-317a-47db-b952-59244215f86c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" podUID="9382dbcf-317a-47db-b952-59244215f86c" Oct 30 13:45:04.378053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.378026 5049 generic.go:356] "Generic (PLEG): container finished" podID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" containerID="ceb0da75b52a0da945cabf160bdbe39cd2a619cbe834e715a46eb6b91d5acf72" exitCode=60 Oct 30 13:45:04.378172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.378107 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" event={"ID":"46fda2f4-2ef1-46c6-a85b-cfce89fe474e","Type":"ContainerDied","Data":"ceb0da75b52a0da945cabf160bdbe39cd2a619cbe834e715a46eb6b91d5acf72"} Oct 30 13:45:04.378412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.378396 5049 scope.go:122] "RemoveContainer" containerID="ceb0da75b52a0da945cabf160bdbe39cd2a619cbe834e715a46eb6b91d5acf72" Oct 30 13:45:04.378702 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:04.378667 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6_odh-ai-gateway-infra(46fda2f4-2ef1-46c6-a85b-cfce89fe474e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" podUID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" Oct 30 13:45:04.380221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.380198 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" event={"ID":"46821723-84da-4e91-b3d1-e4de9f047b93","Type":"ContainerStarted","Data":"89a969315e29a1e7b61340562e886a09af6e183c9c520c5f5ef1a18d76e212a7"} Oct 30 13:45:04.380309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.380244 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" event={"ID":"46821723-84da-4e91-b3d1-e4de9f047b93","Type":"ContainerStarted","Data":"4b587220f0016a6bc6473f24986862092eb47437916725d9e84ae71aa52ed9ba"} Oct 30 13:45:04.381917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.381895 5049 generic.go:356] "Generic (PLEG): container finished" podID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerID="7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5" exitCode=0 Oct 30 13:45:04.382025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.381954 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" Oct 30 13:45:04.382025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.381973 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" event={"ID":"d191137c-94a6-4f18-b4f9-1c2e85de46de","Type":"ContainerDied","Data":"7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5"} Oct 30 13:45:04.382025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.382002 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s" event={"ID":"d191137c-94a6-4f18-b4f9-1c2e85de46de","Type":"ContainerDied","Data":"9d73640b6d5a89e9feda1c3b9971120342d411ef57c614de9669c8abf374ef55"} Oct 30 13:45:04.383751 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.383729 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" event={"ID":"7574c065-ecfa-4afc-b346-2519aa9be554","Type":"ContainerStarted","Data":"6c6ed1b221f96cfb4024c9b922beb87e051999deb842f7eb59f56bc721961c2b"} Oct 30 13:45:04.383868 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.383774 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" event={"ID":"7574c065-ecfa-4afc-b346-2519aa9be554","Type":"ContainerStarted","Data":"74b640c72cd0f8aef777be418237388f6f22fd80c46a6c39b7e31f1abd1d444e"} Oct 30 13:45:04.386392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.386361 5049 generic.go:356] "Generic (PLEG): container finished" podID="e483618c-00a8-458a-b8e8-bc927c60ba52" containerID="348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47" exitCode=0 Oct 30 13:45:04.386484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.386438 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" event={"ID":"e483618c-00a8-458a-b8e8-bc927c60ba52","Type":"ContainerDied","Data":"348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47"} Oct 30 13:45:04.386484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.386462 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" Oct 30 13:45:04.386484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.386473 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp" event={"ID":"e483618c-00a8-458a-b8e8-bc927c60ba52","Type":"ContainerDied","Data":"68d36e9de9045b6f64b5f801eab7d63eedf937763bc760843b3cbd6ee3aa03ec"} Oct 30 13:45:04.389725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.389705 5049 scope.go:122] "RemoveContainer" containerID="ab8606ee8c082b398234f91858886075de48b61365d47180f30315aa768ae585" Oct 30 13:45:04.400187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.400160 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcmgr\" (UniqueName: \"kubernetes.io/projected/d191137c-94a6-4f18-b4f9-1c2e85de46de-kube-api-access-mcmgr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.400187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.400181 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.400187 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.400191 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d191137c-94a6-4f18-b4f9-1c2e85de46de-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.400360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.400199 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.400360 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.400207 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d191137c-94a6-4f18-b4f9-1c2e85de46de-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.405183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.405165 5049 scope.go:122] "RemoveContainer" containerID="7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5" Oct 30 13:45:04.420570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.420544 5049 scope.go:122] "RemoveContainer" containerID="7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5" Oct 30 13:45:04.420946 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:04.420924 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5\": container with ID starting with 7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5 not found: ID does not exist" containerID="7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5" Oct 30 13:45:04.421030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.420955 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5"} err="failed to get container status \"7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5\": rpc error: code = NotFound desc = could not find container \"7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5\": container with ID starting with 7d36139b10271d00a63b4ce1a6ce3716bd1c3677eb5cf33105fd9534eac026a5 not found: ID does not exist" Oct 30 13:45:04.421030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.420976 5049 scope.go:122] "RemoveContainer" containerID="348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47" Oct 30 13:45:04.433475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.433452 5049 scope.go:122] "RemoveContainer" containerID="348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47" Oct 30 13:45:04.433856 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:04.433832 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47\": container with ID starting with 348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47 not found: ID does not exist" containerID="348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47" Oct 30 13:45:04.433966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.433861 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47"} err="failed to get container status \"348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47\": rpc error: code = NotFound desc = could not find container \"348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47\": container with ID starting with 348fe05ac28382d46d207911bc73ffa77cea9151f436bc3ca91422fadc1e0b47 not found: ID does not exist" Oct 30 13:45:04.452848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.452794 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" podStartSLOduration=1.452782247 podStartE2EDuration="1.452782247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.452428009 +0000 UTC m=+3211.267622431" watchObservedRunningTime="2026-10-30 13:45:04.452782247 +0000 UTC m=+3211.267976666" Oct 30 13:45:04.529457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.529403 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" podStartSLOduration=1.5293893650000001 podStartE2EDuration="1.529389365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.526509149 +0000 UTC m=+3211.341703575" watchObservedRunningTime="2026-10-30 13:45:04.529389365 +0000 UTC m=+3211.344583826" Oct 30 13:45:04.561224 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.561140 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s"] Oct 30 13:45:04.569663 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.569621 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-w5-e34fd5-kserve-6978fbb4d9-lxr5s"] Oct 30 13:45:04.595489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.595453 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp"] Oct 30 13:45:04.599172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.599146 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8d82d5-6b4b7886b6-dr6rp"] Oct 30 13:45:04.689191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.689155 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" podUID="ede810a9-3a02-490d-9a25-d0788bea4cdf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:04.689191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:04.689170 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" podUID="409baa7e-092d-4d07-9a46-e611890a8458" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:05.337853 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.337807 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a120e6-66d4-4643-bd63-428f163eaf8c" path="/var/lib/kubelet/pods/20a120e6-66d4-4643-bd63-428f163eaf8c/volumes" Oct 30 13:45:05.338995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.338856 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84de9a2c-5d62-4ab2-a922-360fdffec6ee" path="/var/lib/kubelet/pods/84de9a2c-5d62-4ab2-a922-360fdffec6ee/volumes" Oct 30 13:45:05.339567 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.339542 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8fc97d9-1e7d-4b79-b70e-9036d3dfc610" path="/var/lib/kubelet/pods/a8fc97d9-1e7d-4b79-b70e-9036d3dfc610/volumes" Oct 30 13:45:05.340302 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.340285 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" path="/var/lib/kubelet/pods/d191137c-94a6-4f18-b4f9-1c2e85de46de/volumes" Oct 30 13:45:05.340693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.340678 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e483618c-00a8-458a-b8e8-bc927c60ba52" path="/var/lib/kubelet/pods/e483618c-00a8-458a-b8e8-bc927c60ba52/volumes" Oct 30 13:45:05.398470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.398439 5049 scope.go:122] "RemoveContainer" containerID="abfc09a5c9b6b740307f17e6c71fceebd8d63fbdbddc1303a2db367b36e662c8" Oct 30 13:45:05.398911 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:05.398716 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln_odh-ai-gateway-infra(9382dbcf-317a-47db-b952-59244215f86c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" podUID="9382dbcf-317a-47db-b952-59244215f86c" Oct 30 13:45:05.400123 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.400101 5049 scope.go:122] "RemoveContainer" containerID="ceb0da75b52a0da945cabf160bdbe39cd2a619cbe834e715a46eb6b91d5acf72" Oct 30 13:45:05.400391 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:05.400372 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6_odh-ai-gateway-infra(46fda2f4-2ef1-46c6-a85b-cfce89fe474e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" podUID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" Oct 30 13:45:05.401406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.401387 5049 generic.go:356] "Generic (PLEG): container finished" podID="46821723-84da-4e91-b3d1-e4de9f047b93" containerID="89a969315e29a1e7b61340562e886a09af6e183c9c520c5f5ef1a18d76e212a7" exitCode=0 Oct 30 13:45:05.401477 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.401455 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" event={"ID":"46821723-84da-4e91-b3d1-e4de9f047b93","Type":"ContainerDied","Data":"89a969315e29a1e7b61340562e886a09af6e183c9c520c5f5ef1a18d76e212a7"} Oct 30 13:45:05.404665 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.404642 5049 generic.go:356] "Generic (PLEG): container finished" podID="7574c065-ecfa-4afc-b346-2519aa9be554" containerID="6c6ed1b221f96cfb4024c9b922beb87e051999deb842f7eb59f56bc721961c2b" exitCode=0 Oct 30 13:45:05.404778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.404686 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" event={"ID":"7574c065-ecfa-4afc-b346-2519aa9be554","Type":"ContainerDied","Data":"6c6ed1b221f96cfb4024c9b922beb87e051999deb842f7eb59f56bc721961c2b"} Oct 30 13:45:05.409021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.408994 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k"] Oct 30 13:45:05.409878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.409854 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" Oct 30 13:45:05.409980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.409879 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" Oct 30 13:45:05.409980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.409899 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e483618c-00a8-458a-b8e8-bc927c60ba52" containerName="payload-processing" Oct 30 13:45:05.409980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.409907 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e483618c-00a8-458a-b8e8-bc927c60ba52" containerName="payload-processing" Oct 30 13:45:05.410158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.410033 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d191137c-94a6-4f18-b4f9-1c2e85de46de" containerName="main" Oct 30 13:45:05.410158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.410048 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e483618c-00a8-458a-b8e8-bc927c60ba52" containerName="payload-processing" Oct 30 13:45:05.415393 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.415363 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.435988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.435958 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k"] Oct 30 13:45:05.511778 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.511738 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3feedf6e-fn79k\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.511960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.511868 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hswh8\" (UniqueName: \"kubernetes.io/projected/ffafa367-dab6-48bb-939f-d84a49d43194-kube-api-access-hswh8\") pod \"maas-api-revoke-sub-3feedf6e-fn79k\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.512112 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.512085 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3feedf6e-fn79k\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.544626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.544585 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" podUID="3e86a92d-8512-45b0-8880-ef6432fcf7ff" containerName="payload-processing" containerID="cri-o://05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5" gracePeriod=25 Oct 30 13:45:05.612901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.612808 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3feedf6e-fn79k\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.612901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.612895 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3feedf6e-fn79k\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.613127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.612953 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hswh8\" (UniqueName: \"kubernetes.io/projected/ffafa367-dab6-48bb-939f-d84a49d43194-kube-api-access-hswh8\") pod \"maas-api-revoke-sub-3feedf6e-fn79k\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.613688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.613655 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3feedf6e-fn79k\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.615349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.615303 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3feedf6e-fn79k\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.634584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.634558 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hswh8\" (UniqueName: \"kubernetes.io/projected/ffafa367-dab6-48bb-939f-d84a49d43194-kube-api-access-hswh8\") pod \"maas-api-revoke-sub-3feedf6e-fn79k\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.737117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.737081 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:05.974997 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:05.974915 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:06.046693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.046657 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k"] Oct 30 13:45:06.051186 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:45:06.051156 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffafa367_dab6_48bb_939f_d84a49d43194.slice/crio-cba122de521bd853a7438df7d6eab408af255425968735c5c1e00e9f8658dd48 WatchSource:0}: Error finding container cba122de521bd853a7438df7d6eab408af255425968735c5c1e00e9f8658dd48: Status 404 returned error can't find the container with id cba122de521bd853a7438df7d6eab408af255425968735c5c1e00e9f8658dd48 Oct 30 13:45:06.415714 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.415671 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" event={"ID":"ffafa367-dab6-48bb-939f-d84a49d43194","Type":"ContainerStarted","Data":"54f27db614940c2f010462de057c05b5555547831213694dcecb398f5f9b30d6"} Oct 30 13:45:06.416190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.415743 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" event={"ID":"ffafa367-dab6-48bb-939f-d84a49d43194","Type":"ContainerStarted","Data":"cba122de521bd853a7438df7d6eab408af255425968735c5c1e00e9f8658dd48"} Oct 30 13:45:06.452424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.452346 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" podStartSLOduration=1.452330336 podStartE2EDuration="1.452330336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:06.451925026 +0000 UTC m=+3213.267119453" watchObservedRunningTime="2026-10-30 13:45:06.452330336 +0000 UTC m=+3213.267524753" Oct 30 13:45:06.591293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.591263 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:06.604483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.604456 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:06.725172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.725140 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-xks-ca\") pod \"7574c065-ecfa-4afc-b346-2519aa9be554\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " Oct 30 13:45:06.725353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.725266 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-openshift-ca\") pod \"7574c065-ecfa-4afc-b346-2519aa9be554\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " Oct 30 13:45:06.725353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.725317 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7574c065-ecfa-4afc-b346-2519aa9be554-kube-api-access-wlzwr\" (UniqueName: \"kubernetes.io/projected/7574c065-ecfa-4afc-b346-2519aa9be554-kube-api-access-wlzwr\") pod \"7574c065-ecfa-4afc-b346-2519aa9be554\" (UID: \"7574c065-ecfa-4afc-b346-2519aa9be554\") " Oct 30 13:45:06.725439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.725383 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46821723-84da-4e91-b3d1-e4de9f047b93-kube-api-access-28267\" (UniqueName: \"kubernetes.io/projected/46821723-84da-4e91-b3d1-e4de9f047b93-kube-api-access-28267\") pod \"46821723-84da-4e91-b3d1-e4de9f047b93\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " Oct 30 13:45:06.725439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.725408 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-openshift-ca\") pod \"46821723-84da-4e91-b3d1-e4de9f047b93\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " Oct 30 13:45:06.725541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.725438 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-xks-ca\") pod \"46821723-84da-4e91-b3d1-e4de9f047b93\" (UID: \"46821723-84da-4e91-b3d1-e4de9f047b93\") " Oct 30 13:45:06.725878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.725735 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-openshift-ca" pod "7574c065-ecfa-4afc-b346-2519aa9be554" (UID: "7574c065-ecfa-4afc-b346-2519aa9be554"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:06.725878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.725842 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-openshift-ca" pod "46821723-84da-4e91-b3d1-e4de9f047b93" (UID: "46821723-84da-4e91-b3d1-e4de9f047b93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:06.726092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.725961 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.726092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.725984 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.727900 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.727859 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46821723-84da-4e91-b3d1-e4de9f047b93-kube-api-access-28267" pod "46821723-84da-4e91-b3d1-e4de9f047b93" (UID: "46821723-84da-4e91-b3d1-e4de9f047b93"). InnerVolumeSpecName "kube-api-access-28267". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:06.728284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.728238 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-xks-ca" pod "7574c065-ecfa-4afc-b346-2519aa9be554" (UID: "7574c065-ecfa-4afc-b346-2519aa9be554"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:06.729947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.728360 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7574c065-ecfa-4afc-b346-2519aa9be554-kube-api-access-wlzwr" pod "7574c065-ecfa-4afc-b346-2519aa9be554" (UID: "7574c065-ecfa-4afc-b346-2519aa9be554"). InnerVolumeSpecName "kube-api-access-wlzwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:06.730464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.730438 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-xks-ca" pod "46821723-84da-4e91-b3d1-e4de9f047b93" (UID: "46821723-84da-4e91-b3d1-e4de9f047b93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:06.827252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.827214 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlzwr\" (UniqueName: \"kubernetes.io/projected/7574c065-ecfa-4afc-b346-2519aa9be554-kube-api-access-wlzwr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.827252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.827245 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28267\" (UniqueName: \"kubernetes.io/projected/46821723-84da-4e91-b3d1-e4de9f047b93-kube-api-access-28267\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.827252 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.827255 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46821723-84da-4e91-b3d1-e4de9f047b93-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.827503 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.827267 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7574c065-ecfa-4afc-b346-2519aa9be554-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.917194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:06.917112 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" podUID="3fa7700f-de5f-4302-8147-edb31d44f888" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:07.425093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:07.425059 5049 generic.go:356] "Generic (PLEG): container finished" podID="ffafa367-dab6-48bb-939f-d84a49d43194" containerID="54f27db614940c2f010462de057c05b5555547831213694dcecb398f5f9b30d6" exitCode=0 Oct 30 13:45:07.425549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:07.425154 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" event={"ID":"ffafa367-dab6-48bb-939f-d84a49d43194","Type":"ContainerDied","Data":"54f27db614940c2f010462de057c05b5555547831213694dcecb398f5f9b30d6"} Oct 30 13:45:07.426738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:07.426718 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" Oct 30 13:45:07.426863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:07.426743 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx" event={"ID":"46821723-84da-4e91-b3d1-e4de9f047b93","Type":"ContainerDied","Data":"4b587220f0016a6bc6473f24986862092eb47437916725d9e84ae71aa52ed9ba"} Oct 30 13:45:07.426863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:07.426776 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b587220f0016a6bc6473f24986862092eb47437916725d9e84ae71aa52ed9ba" Oct 30 13:45:07.428030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:07.428006 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" event={"ID":"7574c065-ecfa-4afc-b346-2519aa9be554","Type":"ContainerDied","Data":"74b640c72cd0f8aef777be418237388f6f22fd80c46a6c39b7e31f1abd1d444e"} Oct 30 13:45:07.428030 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:07.428027 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz" Oct 30 13:45:07.428144 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:07.428048 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74b640c72cd0f8aef777be418237388f6f22fd80c46a6c39b7e31f1abd1d444e" Oct 30 13:45:07.824078 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:07.824042 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:08.140833 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.140729 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" containerID="cri-o://92fb3846104009974a50456818e00bd02b6699409c6a197178b7571a4ce667a7" gracePeriod=45 Oct 30 13:45:08.155898 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.155857 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" containerID="cri-o://6ed7b73faaa0fe3b49a4c6561ce8983b52bc42832148677ca8a1a1699b17b3c2" gracePeriod=45 Oct 30 13:45:08.297669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.297624 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused" Oct 30 13:45:08.441195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.441154 5049 generic.go:356] "Generic (PLEG): container finished" podID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerID="6ed7b73faaa0fe3b49a4c6561ce8983b52bc42832148677ca8a1a1699b17b3c2" exitCode=0 Oct 30 13:45:08.441722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.441343 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" event={"ID":"1d97d466-e892-4377-8c31-47dd077dbbc6","Type":"ContainerDied","Data":"6ed7b73faaa0fe3b49a4c6561ce8983b52bc42832148677ca8a1a1699b17b3c2"} Oct 30 13:45:08.443064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.443036 5049 generic.go:356] "Generic (PLEG): container finished" podID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerID="92fb3846104009974a50456818e00bd02b6699409c6a197178b7571a4ce667a7" exitCode=0 Oct 30 13:45:08.443230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.443197 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" event={"ID":"e59bea39-c435-4272-b9e2-ee38897a3f55","Type":"ContainerDied","Data":"92fb3846104009974a50456818e00bd02b6699409c6a197178b7571a4ce667a7"} Oct 30 13:45:08.443433 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.443412 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" event={"ID":"e59bea39-c435-4272-b9e2-ee38897a3f55","Type":"ContainerDied","Data":"ffeebd0a344bb73acdb543630ff207fc22c320c3ae2942f0b6c69235e893747e"} Oct 30 13:45:08.443528 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.443437 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffeebd0a344bb73acdb543630ff207fc22c320c3ae2942f0b6c69235e893747e" Oct 30 13:45:08.447061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.447039 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:45:08.504594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.504552 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:45:08.552860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.552822 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-dshm\") pod \"e59bea39-c435-4272-b9e2-ee38897a3f55\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " Oct 30 13:45:08.553093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.552965 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e59bea39-c435-4272-b9e2-ee38897a3f55-tls-certs\" (UniqueName: \"kubernetes.io/secret/e59bea39-c435-4272-b9e2-ee38897a3f55-tls-certs\") pod \"e59bea39-c435-4272-b9e2-ee38897a3f55\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " Oct 30 13:45:08.553093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.553000 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-tmp-dir\") pod \"e59bea39-c435-4272-b9e2-ee38897a3f55\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " Oct 30 13:45:08.553093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.553028 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-home\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-home\") pod \"e59bea39-c435-4272-b9e2-ee38897a3f55\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " Oct 30 13:45:08.554059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.553129 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-model-cache\") pod \"e59bea39-c435-4272-b9e2-ee38897a3f55\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " Oct 30 13:45:08.554059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.553186 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59bea39-c435-4272-b9e2-ee38897a3f55-kube-api-access-5c96c\" (UniqueName: \"kubernetes.io/projected/e59bea39-c435-4272-b9e2-ee38897a3f55-kube-api-access-5c96c\") pod \"e59bea39-c435-4272-b9e2-ee38897a3f55\" (UID: \"e59bea39-c435-4272-b9e2-ee38897a3f55\") " Oct 30 13:45:08.555007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.554893 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-tmp-dir" pod "e59bea39-c435-4272-b9e2-ee38897a3f55" (UID: "e59bea39-c435-4272-b9e2-ee38897a3f55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:08.555007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.554966 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-model-cache" pod "e59bea39-c435-4272-b9e2-ee38897a3f55" (UID: "e59bea39-c435-4272-b9e2-ee38897a3f55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:08.557153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.557128 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-home" pod "e59bea39-c435-4272-b9e2-ee38897a3f55" (UID: "e59bea39-c435-4272-b9e2-ee38897a3f55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:08.560460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.560092 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-dshm" pod "e59bea39-c435-4272-b9e2-ee38897a3f55" (UID: "e59bea39-c435-4272-b9e2-ee38897a3f55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:08.560460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.560136 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59bea39-c435-4272-b9e2-ee38897a3f55-kube-api-access-5c96c" pod "e59bea39-c435-4272-b9e2-ee38897a3f55" (UID: "e59bea39-c435-4272-b9e2-ee38897a3f55"). InnerVolumeSpecName "kube-api-access-5c96c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:08.560680 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.560601 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e59bea39-c435-4272-b9e2-ee38897a3f55-tls-certs" pod "e59bea39-c435-4272-b9e2-ee38897a3f55" (UID: "e59bea39-c435-4272-b9e2-ee38897a3f55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:08.624254 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.624227 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:08.654876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.654845 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-home\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-home\") pod \"1d97d466-e892-4377-8c31-47dd077dbbc6\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " Oct 30 13:45:08.655034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.654926 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-model-cache\") pod \"1d97d466-e892-4377-8c31-47dd077dbbc6\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " Oct 30 13:45:08.655034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.654977 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d97d466-e892-4377-8c31-47dd077dbbc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d97d466-e892-4377-8c31-47dd077dbbc6-tls-certs\") pod \"1d97d466-e892-4377-8c31-47dd077dbbc6\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " Oct 30 13:45:08.655034 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.655028 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-tmp-dir\") pod \"1d97d466-e892-4377-8c31-47dd077dbbc6\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " Oct 30 13:45:08.655239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.655085 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d97d466-e892-4377-8c31-47dd077dbbc6-kube-api-access-9hxtx\" (UniqueName: \"kubernetes.io/projected/1d97d466-e892-4377-8c31-47dd077dbbc6-kube-api-access-9hxtx\") pod \"1d97d466-e892-4377-8c31-47dd077dbbc6\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " Oct 30 13:45:08.655239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.655098 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-home" pod "1d97d466-e892-4377-8c31-47dd077dbbc6" (UID: "1d97d466-e892-4377-8c31-47dd077dbbc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:08.655239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.655131 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-dshm\") pod \"1d97d466-e892-4377-8c31-47dd077dbbc6\" (UID: \"1d97d466-e892-4377-8c31-47dd077dbbc6\") " Oct 30 13:45:08.655416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.655302 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-model-cache" pod "1d97d466-e892-4377-8c31-47dd077dbbc6" (UID: "1d97d466-e892-4377-8c31-47dd077dbbc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:08.655976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.655935 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.655976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.655967 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.655976 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.655980 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c96c\" (UniqueName: \"kubernetes.io/projected/e59bea39-c435-4272-b9e2-ee38897a3f55-kube-api-access-5c96c\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.656246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.655992 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.656246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.656007 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.656246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.656018 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e59bea39-c435-4272-b9e2-ee38897a3f55-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.656246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.656030 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.656246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.656041 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e59bea39-c435-4272-b9e2-ee38897a3f55-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.656544 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.656267 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-tmp-dir" pod "1d97d466-e892-4377-8c31-47dd077dbbc6" (UID: "1d97d466-e892-4377-8c31-47dd077dbbc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:08.658107 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.658085 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d97d466-e892-4377-8c31-47dd077dbbc6-kube-api-access-9hxtx" pod "1d97d466-e892-4377-8c31-47dd077dbbc6" (UID: "1d97d466-e892-4377-8c31-47dd077dbbc6"). InnerVolumeSpecName "kube-api-access-9hxtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:08.658248 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.658109 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d97d466-e892-4377-8c31-47dd077dbbc6-tls-certs" pod "1d97d466-e892-4377-8c31-47dd077dbbc6" (UID: "1d97d466-e892-4377-8c31-47dd077dbbc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:08.658698 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.658678 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-dshm" pod "1d97d466-e892-4377-8c31-47dd077dbbc6" (UID: "1d97d466-e892-4377-8c31-47dd077dbbc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:08.756447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.756352 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-xks-ca\") pod \"ffafa367-dab6-48bb-939f-d84a49d43194\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " Oct 30 13:45:08.756447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.756407 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffafa367-dab6-48bb-939f-d84a49d43194-kube-api-access-hswh8\" (UniqueName: \"kubernetes.io/projected/ffafa367-dab6-48bb-939f-d84a49d43194-kube-api-access-hswh8\") pod \"ffafa367-dab6-48bb-939f-d84a49d43194\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " Oct 30 13:45:08.756678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.756508 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-openshift-ca\") pod \"ffafa367-dab6-48bb-939f-d84a49d43194\" (UID: \"ffafa367-dab6-48bb-939f-d84a49d43194\") " Oct 30 13:45:08.756828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.756806 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.756883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.756834 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hxtx\" (UniqueName: \"kubernetes.io/projected/1d97d466-e892-4377-8c31-47dd077dbbc6-kube-api-access-9hxtx\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.756883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.756849 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d97d466-e892-4377-8c31-47dd077dbbc6-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.756883 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.756863 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d97d466-e892-4377-8c31-47dd077dbbc6-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.757036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.756990 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-openshift-ca" pod "ffafa367-dab6-48bb-939f-d84a49d43194" (UID: "ffafa367-dab6-48bb-939f-d84a49d43194"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:08.758754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.758721 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffafa367-dab6-48bb-939f-d84a49d43194-kube-api-access-hswh8" pod "ffafa367-dab6-48bb-939f-d84a49d43194" (UID: "ffafa367-dab6-48bb-939f-d84a49d43194"). InnerVolumeSpecName "kube-api-access-hswh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:08.759124 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.759092 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-xks-ca" pod "ffafa367-dab6-48bb-939f-d84a49d43194" (UID: "ffafa367-dab6-48bb-939f-d84a49d43194"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:08.857645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.857608 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hswh8\" (UniqueName: \"kubernetes.io/projected/ffafa367-dab6-48bb-939f-d84a49d43194-kube-api-access-hswh8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.857645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.857638 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:08.857645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:08.857648 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffafa367-dab6-48bb-939f-d84a49d43194-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:09.454212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.454124 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" Oct 30 13:45:09.454212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.454172 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r" event={"ID":"1d97d466-e892-4377-8c31-47dd077dbbc6","Type":"ContainerDied","Data":"1df40a181fcc193a53dd77ad5342b821a82acfc787f04664948e0f61c3783443"} Oct 30 13:45:09.454842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.454233 5049 scope.go:122] "RemoveContainer" containerID="6ed7b73faaa0fe3b49a4c6561ce8983b52bc42832148677ca8a1a1699b17b3c2" Oct 30 13:45:09.456311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.456280 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" Oct 30 13:45:09.456311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.456304 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn" Oct 30 13:45:09.456487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.456362 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k" event={"ID":"ffafa367-dab6-48bb-939f-d84a49d43194","Type":"ContainerDied","Data":"cba122de521bd853a7438df7d6eab408af255425968735c5c1e00e9f8658dd48"} Oct 30 13:45:09.456487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.456402 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cba122de521bd853a7438df7d6eab408af255425968735c5c1e00e9f8658dd48" Oct 30 13:45:09.516496 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.516454 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn"] Oct 30 13:45:09.521954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.521921 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/premium-w0-f07dfe-kserve-98dff7894-xnchn"] Oct 30 13:45:09.552699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.552643 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r"] Oct 30 13:45:09.555852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.555826 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/simulator-w0-f07dfe-kserve-848887bb7f-6r62r"] Oct 30 13:45:09.632060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.632026 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" podUID="3e86a92d-8512-45b0-8880-ef6432fcf7ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:09.684706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.684675 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:09.759640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.759550 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" podUID="91e5f412-4939-44cd-937d-2bf38707178e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:09.759640 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:09.759559 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" podUID="55f9f0d5-52a1-434f-acb9-fdbfe8671335" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:10.013634 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.013544 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" containerID="cri-o://6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e" gracePeriod=45 Oct 30 13:45:10.295595 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.295568 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:45:10.373351 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.373296 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-home\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-home\") pod \"ea6ab97e-bb42-458a-8a78-f8752042da2e\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " Oct 30 13:45:10.373545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.373369 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6ab97e-bb42-458a-8a78-f8752042da2e-kube-api-access-hld64\" (UniqueName: \"kubernetes.io/projected/ea6ab97e-bb42-458a-8a78-f8752042da2e-kube-api-access-hld64\") pod \"ea6ab97e-bb42-458a-8a78-f8752042da2e\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " Oct 30 13:45:10.373545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.373507 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-tmp-dir\") pod \"ea6ab97e-bb42-458a-8a78-f8752042da2e\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " Oct 30 13:45:10.373545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.373539 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-model-cache\") pod \"ea6ab97e-bb42-458a-8a78-f8752042da2e\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " Oct 30 13:45:10.373747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.373573 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea6ab97e-bb42-458a-8a78-f8752042da2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6ab97e-bb42-458a-8a78-f8752042da2e-tls-certs\") pod \"ea6ab97e-bb42-458a-8a78-f8752042da2e\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " Oct 30 13:45:10.373747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.373577 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-home" pod "ea6ab97e-bb42-458a-8a78-f8752042da2e" (UID: "ea6ab97e-bb42-458a-8a78-f8752042da2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.373747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.373666 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-dshm\") pod \"ea6ab97e-bb42-458a-8a78-f8752042da2e\" (UID: \"ea6ab97e-bb42-458a-8a78-f8752042da2e\") " Oct 30 13:45:10.373912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.373817 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-model-cache" pod "ea6ab97e-bb42-458a-8a78-f8752042da2e" (UID: "ea6ab97e-bb42-458a-8a78-f8752042da2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.373912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.373799 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-tmp-dir" pod "ea6ab97e-bb42-458a-8a78-f8752042da2e" (UID: "ea6ab97e-bb42-458a-8a78-f8752042da2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.374116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.374095 5049 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-home\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.374169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.374123 5049 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-tmp-dir\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.374169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.374138 5049 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-model-cache\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.376195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.376165 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-dshm" pod "ea6ab97e-bb42-458a-8a78-f8752042da2e" (UID: "ea6ab97e-bb42-458a-8a78-f8752042da2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.376385 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.376365 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6ab97e-bb42-458a-8a78-f8752042da2e-kube-api-access-hld64" pod "ea6ab97e-bb42-458a-8a78-f8752042da2e" (UID: "ea6ab97e-bb42-458a-8a78-f8752042da2e"). InnerVolumeSpecName "kube-api-access-hld64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:10.376693 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.376672 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea6ab97e-bb42-458a-8a78-f8752042da2e-tls-certs" pod "ea6ab97e-bb42-458a-8a78-f8752042da2e" (UID: "ea6ab97e-bb42-458a-8a78-f8752042da2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:10.466418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.466373 5049 generic.go:356] "Generic (PLEG): container finished" podID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerID="6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e" exitCode=0 Oct 30 13:45:10.466832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.466424 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" Oct 30 13:45:10.466832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.466453 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" event={"ID":"ea6ab97e-bb42-458a-8a78-f8752042da2e","Type":"ContainerDied","Data":"6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e"} Oct 30 13:45:10.466832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.466500 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m" event={"ID":"ea6ab97e-bb42-458a-8a78-f8752042da2e","Type":"ContainerDied","Data":"af08d9cb677773c346afe9d851d3bb26e601a8da533ac96e99899bc5d17401f9"} Oct 30 13:45:10.466832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.466521 5049 scope.go:122] "RemoveContainer" containerID="6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e" Oct 30 13:45:10.474985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.474842 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hld64\" (UniqueName: \"kubernetes.io/projected/ea6ab97e-bb42-458a-8a78-f8752042da2e-kube-api-access-hld64\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.474985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.474867 5049 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6ab97e-bb42-458a-8a78-f8752042da2e-tls-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.474985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.474884 5049 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6ab97e-bb42-458a-8a78-f8752042da2e-dshm\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.478147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.478127 5049 scope.go:122] "RemoveContainer" containerID="6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e" Oct 30 13:45:10.478432 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:10.478412 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e\": container with ID starting with 6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e not found: ID does not exist" containerID="6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e" Oct 30 13:45:10.478489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.478441 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e"} err="failed to get container status \"6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e\": rpc error: code = NotFound desc = could not find container \"6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e\": container with ID starting with 6633779327f7f85eab5f3cc491fcc0dc8cd25697f826e8f7a05dc62ee640153e not found: ID does not exist" Oct 30 13:45:10.514042 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.514009 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m"] Oct 30 13:45:10.517931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:10.517898 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/premium-w1-52a649-kserve-58ff6dc548-bxt4m"] Oct 30 13:45:11.238972 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:11.238912 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:11.334894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:11.334860 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" path="/var/lib/kubelet/pods/1d97d466-e892-4377-8c31-47dd077dbbc6/volumes" Oct 30 13:45:11.335403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:11.335382 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" path="/var/lib/kubelet/pods/e59bea39-c435-4272-b9e2-ee38897a3f55/volumes" Oct 30 13:45:11.335845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:11.335827 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" path="/var/lib/kubelet/pods/ea6ab97e-bb42-458a-8a78-f8752042da2e/volumes" Oct 30 13:45:11.557500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:11.557457 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:11.606850 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:11.606814 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q"] Oct 30 13:45:11.607073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:11.607048 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" podUID="fd91c2b7-2153-434b-8a1c-817eabac78b0" containerName="istio-proxy" containerID="cri-o://27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579" gracePeriod=30 Oct 30 13:45:11.732722 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:11.732687 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:12.132025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.131991 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:12.245886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.245853 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj"] Oct 30 13:45:12.246472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246456 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ffafa367-dab6-48bb-939f-d84a49d43194" containerName="revoke-keys" Oct 30 13:45:12.246534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246474 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffafa367-dab6-48bb-939f-d84a49d43194" containerName="revoke-keys" Oct 30 13:45:12.246534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246486 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" Oct 30 13:45:12.246534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246494 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" Oct 30 13:45:12.246534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246505 5049 cpu_manager.go:414] "containerMap: removing container" podUID="46821723-84da-4e91-b3d1-e4de9f047b93" containerName="revoke-keys" Oct 30 13:45:12.246534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246511 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46821723-84da-4e91-b3d1-e4de9f047b93" containerName="revoke-keys" Oct 30 13:45:12.246534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246527 5049 cpu_manager.go:414] "containerMap: removing container" podUID="7574c065-ecfa-4afc-b346-2519aa9be554" containerName="revoke-keys" Oct 30 13:45:12.246534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246534 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7574c065-ecfa-4afc-b346-2519aa9be554" containerName="revoke-keys" Oct 30 13:45:12.246768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246541 5049 cpu_manager.go:414] "containerMap: removing container" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" Oct 30 13:45:12.246768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246546 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" Oct 30 13:45:12.246768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246553 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" Oct 30 13:45:12.246768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246558 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" Oct 30 13:45:12.246768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246660 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffafa367-dab6-48bb-939f-d84a49d43194" containerName="revoke-keys" Oct 30 13:45:12.246768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246668 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6ab97e-bb42-458a-8a78-f8752042da2e" containerName="main" Oct 30 13:45:12.246768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246674 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="46821723-84da-4e91-b3d1-e4de9f047b93" containerName="revoke-keys" Oct 30 13:45:12.246768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246682 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="7574c065-ecfa-4afc-b346-2519aa9be554" containerName="revoke-keys" Oct 30 13:45:12.246768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246688 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d97d466-e892-4377-8c31-47dd077dbbc6" containerName="main" Oct 30 13:45:12.246768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.246694 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59bea39-c435-4272-b9e2-ee38897a3f55" containerName="main" Oct 30 13:45:12.251484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.251463 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.274263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.274236 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj"] Oct 30 13:45:12.280805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.280773 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:45:12.394389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.394294 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5ww7\" (UniqueName: \"kubernetes.io/projected/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-kube-api-access-s5ww7\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.394529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.394412 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.394529 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.394491 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.495844 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.495803 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.496022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.495937 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5ww7\" (UniqueName: \"kubernetes.io/projected/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-kube-api-access-s5ww7\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.496022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.495973 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.496584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.496558 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.498549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.498525 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.520945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.520921 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5ww7\" (UniqueName: \"kubernetes.io/projected/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-kube-api-access-s5ww7\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.565990 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.565957 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:12.871570 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:12.871538 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj"] Oct 30 13:45:12.875429 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:45:12.875398 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7851a9ca_d13b_4a9f_ab1a_9e86e44696aa.slice/crio-e8e8d21bc1572916703a13483579b296c3393b22babeb7fafd90c25882eae4df WatchSource:0}: Error finding container e8e8d21bc1572916703a13483579b296c3393b22babeb7fafd90c25882eae4df: Status 404 returned error can't find the container with id e8e8d21bc1572916703a13483579b296c3393b22babeb7fafd90c25882eae4df Oct 30 13:45:13.462138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.462038 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6"] Oct 30 13:45:13.487533 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.487496 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:13.494260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.493522 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" event={"ID":"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa","Type":"ContainerStarted","Data":"dd9c07d4b999eff663e24fba7c9c0434b874d027422642f7006c14fff305409e"} Oct 30 13:45:13.494260 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.493585 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" event={"ID":"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa","Type":"ContainerStarted","Data":"e8e8d21bc1572916703a13483579b296c3393b22babeb7fafd90c25882eae4df"} Oct 30 13:45:13.540291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.540237 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" podStartSLOduration=1.54021879 podStartE2EDuration="1.54021879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:13.539384858 +0000 UTC m=+3220.354579285" watchObservedRunningTime="2026-10-30 13:45:13.54021879 +0000 UTC m=+3220.355413212" Oct 30 13:45:13.626339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.626302 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" Oct 30 13:45:13.718097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.718017 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-kube-api-access-szcbr\" (UniqueName: \"kubernetes.io/projected/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-kube-api-access-szcbr\") pod \"46fda2f4-2ef1-46c6-a85b-cfce89fe474e\" (UID: \"46fda2f4-2ef1-46c6-a85b-cfce89fe474e\") " Oct 30 13:45:13.718261 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.718131 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-maas-api-service-ca\") pod \"46fda2f4-2ef1-46c6-a85b-cfce89fe474e\" (UID: \"46fda2f4-2ef1-46c6-a85b-cfce89fe474e\") " Oct 30 13:45:13.718554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.718532 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-maas-api-service-ca" pod "46fda2f4-2ef1-46c6-a85b-cfce89fe474e" (UID: "46fda2f4-2ef1-46c6-a85b-cfce89fe474e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:13.720146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.720123 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-kube-api-access-szcbr" pod "46fda2f4-2ef1-46c6-a85b-cfce89fe474e" (UID: "46fda2f4-2ef1-46c6-a85b-cfce89fe474e"). InnerVolumeSpecName "kube-api-access-szcbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:13.819588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.819544 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szcbr\" (UniqueName: \"kubernetes.io/projected/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-kube-api-access-szcbr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:13.819588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:13.819585 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fda2f4-2ef1-46c6-a85b-cfce89fe474e-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:14.176093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.176007 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:14.340726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.340694 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:14.503808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.503726 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" Oct 30 13:45:14.503808 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.503748 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6" event={"ID":"46fda2f4-2ef1-46c6-a85b-cfce89fe474e","Type":"ContainerDied","Data":"8c09dfdbb4a18d5b50ab2039b4692677c475c1229c2864bc86389f31b6eaa08a"} Oct 30 13:45:14.504038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.503810 5049 scope.go:122] "RemoveContainer" containerID="ceb0da75b52a0da945cabf160bdbe39cd2a619cbe834e715a46eb6b91d5acf72" Oct 30 13:45:14.505791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.505764 5049 generic.go:356] "Generic (PLEG): container finished" podID="7851a9ca-d13b-4a9f-ab1a-9e86e44696aa" containerID="dd9c07d4b999eff663e24fba7c9c0434b874d027422642f7006c14fff305409e" exitCode=0 Oct 30 13:45:14.505937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.505842 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" event={"ID":"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa","Type":"ContainerDied","Data":"dd9c07d4b999eff663e24fba7c9c0434b874d027422642f7006c14fff305409e"} Oct 30 13:45:14.590448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.590411 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6"] Oct 30 13:45:14.597334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.597282 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889465-fj2q6"] Oct 30 13:45:14.689116 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.689077 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" podUID="ede810a9-3a02-490d-9a25-d0788bea4cdf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:14.689617 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.689086 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" podUID="409baa7e-092d-4d07-9a46-e611890a8458" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:14.811178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:14.811142 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:15.328079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.328046 5049 scope.go:122] "RemoveContainer" containerID="abfc09a5c9b6b740307f17e6c71fceebd8d63fbdbddc1303a2db367b36e662c8" Oct 30 13:45:15.335242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.335217 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" path="/var/lib/kubelet/pods/46fda2f4-2ef1-46c6-a85b-cfce89fe474e/volumes" Oct 30 13:45:15.669583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.669559 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:15.840822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.840781 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-xks-ca\") pod \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " Oct 30 13:45:15.841303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.840946 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-kube-api-access-s5ww7\" (UniqueName: \"kubernetes.io/projected/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-kube-api-access-s5ww7\") pod \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " Oct 30 13:45:15.841303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.841026 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-openshift-ca\") pod \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\" (UID: \"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa\") " Oct 30 13:45:15.841460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.841389 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-openshift-ca" pod "7851a9ca-d13b-4a9f-ab1a-9e86e44696aa" (UID: "7851a9ca-d13b-4a9f-ab1a-9e86e44696aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:15.843159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.843125 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-kube-api-access-s5ww7" pod "7851a9ca-d13b-4a9f-ab1a-9e86e44696aa" (UID: "7851a9ca-d13b-4a9f-ab1a-9e86e44696aa"). InnerVolumeSpecName "kube-api-access-s5ww7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:15.843396 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.843372 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-xks-ca" pod "7851a9ca-d13b-4a9f-ab1a-9e86e44696aa" (UID: "7851a9ca-d13b-4a9f-ab1a-9e86e44696aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:15.942342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.942239 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5ww7\" (UniqueName: \"kubernetes.io/projected/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-kube-api-access-s5ww7\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:15.942342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.942271 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:15.942342 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:15.942285 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.054756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.054719 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:16.530785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.530652 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" event={"ID":"9382dbcf-317a-47db-b952-59244215f86c","Type":"ContainerDied","Data":"ab4bf42b8b620d0fecd5e0afef42e4ac9ef06ad6a6ef565b6dc6dae7e4afb347"} Oct 30 13:45:16.530967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.530848 5049 scope.go:122] "RemoveContainer" containerID="abfc09a5c9b6b740307f17e6c71fceebd8d63fbdbddc1303a2db367b36e662c8" Oct 30 13:45:16.531495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.531066 5049 generic.go:356] "Generic (PLEG): container finished" podID="9382dbcf-317a-47db-b952-59244215f86c" containerID="ab4bf42b8b620d0fecd5e0afef42e4ac9ef06ad6a6ef565b6dc6dae7e4afb347" exitCode=60 Oct 30 13:45:16.531652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.531556 5049 scope.go:122] "RemoveContainer" containerID="ab4bf42b8b620d0fecd5e0afef42e4ac9ef06ad6a6ef565b6dc6dae7e4afb347" Oct 30 13:45:16.538997 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:16.532027 5049 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln_odh-ai-gateway-infra(9382dbcf-317a-47db-b952-59244215f86c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" podUID="9382dbcf-317a-47db-b952-59244215f86c" Oct 30 13:45:16.542877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.542855 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" Oct 30 13:45:16.543090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.542868 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj" event={"ID":"7851a9ca-d13b-4a9f-ab1a-9e86e44696aa","Type":"ContainerDied","Data":"e8e8d21bc1572916703a13483579b296c3393b22babeb7fafd90c25882eae4df"} Oct 30 13:45:16.543218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.543117 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8e8d21bc1572916703a13483579b296c3393b22babeb7fafd90c25882eae4df" Oct 30 13:45:16.648995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.648959 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:16.767007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.766973 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp"] Oct 30 13:45:16.770974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.770941 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52a649-jn2jp"] Oct 30 13:45:16.807806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.807774 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:16.891111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.891083 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:45:16.916293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:16.916254 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" podUID="3fa7700f-de5f-4302-8147-edb31d44f888" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:17.052955 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.052924 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd91c2b7-2153-434b-8a1c-817eabac78b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd91c2b7-2153-434b-8a1c-817eabac78b0-istiod-ca-cert\") pod \"fd91c2b7-2153-434b-8a1c-817eabac78b0\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " Oct 30 13:45:17.053136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.052987 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-socket\") pod \"fd91c2b7-2153-434b-8a1c-817eabac78b0\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " Oct 30 13:45:17.053136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053014 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-credential-socket\") pod \"fd91c2b7-2153-434b-8a1c-817eabac78b0\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " Oct 30 13:45:17.053136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053035 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-data\") pod \"fd91c2b7-2153-434b-8a1c-817eabac78b0\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " Oct 30 13:45:17.053136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053054 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-certs\") pod \"fd91c2b7-2153-434b-8a1c-817eabac78b0\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " Oct 30 13:45:17.053136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053106 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-kube-api-access-dw9bw\" (UniqueName: \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-kube-api-access-dw9bw\") pod \"fd91c2b7-2153-434b-8a1c-817eabac78b0\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " Oct 30 13:45:17.053136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053135 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-podinfo\") pod \"fd91c2b7-2153-434b-8a1c-817eabac78b0\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " Oct 30 13:45:17.053594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053211 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-envoy\") pod \"fd91c2b7-2153-434b-8a1c-817eabac78b0\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " Oct 30 13:45:17.053594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053244 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-socket" pod "fd91c2b7-2153-434b-8a1c-817eabac78b0" (UID: "fd91c2b7-2153-434b-8a1c-817eabac78b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:17.053594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053342 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-token\" (UniqueName: \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-token\") pod \"fd91c2b7-2153-434b-8a1c-817eabac78b0\" (UID: \"fd91c2b7-2153-434b-8a1c-817eabac78b0\") " Oct 30 13:45:17.053594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053507 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-credential-socket" pod "fd91c2b7-2153-434b-8a1c-817eabac78b0" (UID: "fd91c2b7-2153-434b-8a1c-817eabac78b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:17.053594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053503 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd91c2b7-2153-434b-8a1c-817eabac78b0-istiod-ca-cert" pod "fd91c2b7-2153-434b-8a1c-817eabac78b0" (UID: "fd91c2b7-2153-434b-8a1c-817eabac78b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:17.053860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053750 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd91c2b7-2153-434b-8a1c-817eabac78b0-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.053860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053771 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.053860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053787 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.054096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.053870 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-data" pod "fd91c2b7-2153-434b-8a1c-817eabac78b0" (UID: "fd91c2b7-2153-434b-8a1c-817eabac78b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:17.054165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.054134 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-certs" pod "fd91c2b7-2153-434b-8a1c-817eabac78b0" (UID: "fd91c2b7-2153-434b-8a1c-817eabac78b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:17.055926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.055904 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-podinfo" pod "fd91c2b7-2153-434b-8a1c-817eabac78b0" (UID: "fd91c2b7-2153-434b-8a1c-817eabac78b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:17.056111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.056085 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-envoy" pod "fd91c2b7-2153-434b-8a1c-817eabac78b0" (UID: "fd91c2b7-2153-434b-8a1c-817eabac78b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:17.056111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.056095 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-token" pod "fd91c2b7-2153-434b-8a1c-817eabac78b0" (UID: "fd91c2b7-2153-434b-8a1c-817eabac78b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:17.056229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.056199 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-kube-api-access-dw9bw" pod "fd91c2b7-2153-434b-8a1c-817eabac78b0" (UID: "fd91c2b7-2153-434b-8a1c-817eabac78b0"). InnerVolumeSpecName "kube-api-access-dw9bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:17.154797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.154760 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.154797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.154790 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.154797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.154799 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.154797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.154807 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd91c2b7-2153-434b-8a1c-817eabac78b0-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.155080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.154815 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9bw\" (UniqueName: \"kubernetes.io/projected/fd91c2b7-2153-434b-8a1c-817eabac78b0-kube-api-access-dw9bw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.155080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.154825 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd91c2b7-2153-434b-8a1c-817eabac78b0-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.293848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.293810 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:17.335671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.335584 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1" path="/var/lib/kubelet/pods/40a09da6-0d7c-4c9c-b11e-1ee66eaed1a1/volumes" Oct 30 13:45:17.485954 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.485915 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:17.553842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.553803 5049 generic.go:356] "Generic (PLEG): container finished" podID="fd91c2b7-2153-434b-8a1c-817eabac78b0" containerID="27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579" exitCode=0 Oct 30 13:45:17.554061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.553848 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" Oct 30 13:45:17.554061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.553883 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" event={"ID":"fd91c2b7-2153-434b-8a1c-817eabac78b0","Type":"ContainerDied","Data":"27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579"} Oct 30 13:45:17.554061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.553952 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q" event={"ID":"fd91c2b7-2153-434b-8a1c-817eabac78b0","Type":"ContainerDied","Data":"1c2796481a3ca27f7b03c4a12e353352b3dafb3fbb9442d15ee6e8145af183d9"} Oct 30 13:45:17.554061 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.553981 5049 scope.go:122] "RemoveContainer" containerID="27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579" Oct 30 13:45:17.566098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.566078 5049 scope.go:122] "RemoveContainer" containerID="27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579" Oct 30 13:45:17.566426 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:17.566403 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579\": container with ID starting with 27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579 not found: ID does not exist" containerID="27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579" Oct 30 13:45:17.566514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.566434 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579"} err="failed to get container status \"27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579\": rpc error: code = NotFound desc = could not find container \"27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579\": container with ID starting with 27d0add71347116c347008924cf4c4530e9855c226cda74fdbd50acea132f579 not found: ID does not exist" Oct 30 13:45:17.581854 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.581824 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln"] Oct 30 13:45:17.612886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.612781 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q"] Oct 30 13:45:17.617282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.617253 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-lb98q"] Oct 30 13:45:17.744588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.744559 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" Oct 30 13:45:17.863208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.863118 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9382dbcf-317a-47db-b952-59244215f86c-kube-api-access-rhdbp\" (UniqueName: \"kubernetes.io/projected/9382dbcf-317a-47db-b952-59244215f86c-kube-api-access-rhdbp\") pod \"9382dbcf-317a-47db-b952-59244215f86c\" (UID: \"9382dbcf-317a-47db-b952-59244215f86c\") " Oct 30 13:45:17.863208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.863177 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9382dbcf-317a-47db-b952-59244215f86c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9382dbcf-317a-47db-b952-59244215f86c-maas-api-service-ca\") pod \"9382dbcf-317a-47db-b952-59244215f86c\" (UID: \"9382dbcf-317a-47db-b952-59244215f86c\") " Oct 30 13:45:17.863633 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.863596 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9382dbcf-317a-47db-b952-59244215f86c-maas-api-service-ca" pod "9382dbcf-317a-47db-b952-59244215f86c" (UID: "9382dbcf-317a-47db-b952-59244215f86c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:17.865242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.865217 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9382dbcf-317a-47db-b952-59244215f86c-kube-api-access-rhdbp" pod "9382dbcf-317a-47db-b952-59244215f86c" (UID: "9382dbcf-317a-47db-b952-59244215f86c"). InnerVolumeSpecName "kube-api-access-rhdbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:17.964467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.964434 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9382dbcf-317a-47db-b952-59244215f86c-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.964467 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:17.964463 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhdbp\" (UniqueName: \"kubernetes.io/projected/9382dbcf-317a-47db-b952-59244215f86c-kube-api-access-rhdbp\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:18.568565 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:18.568536 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" Oct 30 13:45:18.568755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:18.568534 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln" event={"ID":"9382dbcf-317a-47db-b952-59244215f86c","Type":"ContainerDied","Data":"5b74c92e424c4de8e68b86368ca2632cdf650ecd7011c938c4ceae9847e548b7"} Oct 30 13:45:18.568755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:18.568666 5049 scope.go:122] "RemoveContainer" containerID="ab4bf42b8b620d0fecd5e0afef42e4ac9ef06ad6a6ef565b6dc6dae7e4afb347" Oct 30 13:45:18.647523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:18.647485 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln"] Oct 30 13:45:18.657706 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:18.657676 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889465-fdnln"] Oct 30 13:45:18.695702 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:18.695672 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:19.266796 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:19.266756 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:19.335317 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:19.335283 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9382dbcf-317a-47db-b952-59244215f86c" path="/var/lib/kubelet/pods/9382dbcf-317a-47db-b952-59244215f86c/volumes" Oct 30 13:45:19.335785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:19.335768 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd91c2b7-2153-434b-8a1c-817eabac78b0" path="/var/lib/kubelet/pods/fd91c2b7-2153-434b-8a1c-817eabac78b0/volumes" Oct 30 13:45:19.363867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:19.363833 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq"] Oct 30 13:45:19.373768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:19.373737 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e34fd5-xr5jq"] Oct 30 13:45:19.420987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:19.420937 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:19.632133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:19.632086 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" podUID="3e86a92d-8512-45b0-8880-ef6432fcf7ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:19.759662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:19.759625 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" podUID="91e5f412-4939-44cd-937d-2bf38707178e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:19.759816 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:19.759716 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" podUID="55f9f0d5-52a1-434f-acb9-fdbfe8671335" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:19.935878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:19.934946 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:20.006916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.006887 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:45:20.083336 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.083278 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa7700f-de5f-4302-8147-edb31d44f888-kube-api-access-qztr6\" (UniqueName: \"kubernetes.io/projected/3fa7700f-de5f-4302-8147-edb31d44f888-kube-api-access-qztr6\") pod \"3fa7700f-de5f-4302-8147-edb31d44f888\" (UID: \"3fa7700f-de5f-4302-8147-edb31d44f888\") " Oct 30 13:45:20.083549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.083374 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa7700f-de5f-4302-8147-edb31d44f888-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fa7700f-de5f-4302-8147-edb31d44f888-plugins-config-volume\") pod \"3fa7700f-de5f-4302-8147-edb31d44f888\" (UID: \"3fa7700f-de5f-4302-8147-edb31d44f888\") " Oct 30 13:45:20.084018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.083985 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa7700f-de5f-4302-8147-edb31d44f888-plugins-config-volume" pod "3fa7700f-de5f-4302-8147-edb31d44f888" (UID: "3fa7700f-de5f-4302-8147-edb31d44f888"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:20.086314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.086284 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa7700f-de5f-4302-8147-edb31d44f888-kube-api-access-qztr6" pod "3fa7700f-de5f-4302-8147-edb31d44f888" (UID: "3fa7700f-de5f-4302-8147-edb31d44f888"). InnerVolumeSpecName "kube-api-access-qztr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:20.184775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.184697 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qztr6\" (UniqueName: \"kubernetes.io/projected/3fa7700f-de5f-4302-8147-edb31d44f888-kube-api-access-qztr6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.184775 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.184727 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fa7700f-de5f-4302-8147-edb31d44f888-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.585469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.585426 5049 generic.go:356] "Generic (PLEG): container finished" podID="3fa7700f-de5f-4302-8147-edb31d44f888" containerID="e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6" exitCode=0 Oct 30 13:45:20.585469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.585470 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" Oct 30 13:45:20.585979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.585492 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" event={"ID":"3fa7700f-de5f-4302-8147-edb31d44f888","Type":"ContainerDied","Data":"e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6"} Oct 30 13:45:20.585979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.585551 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2" event={"ID":"3fa7700f-de5f-4302-8147-edb31d44f888","Type":"ContainerDied","Data":"a3019bfc4bdc00e837950343b35b48f824ad1c081673d86589c0d2579d482a69"} Oct 30 13:45:20.585979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.585574 5049 scope.go:122] "RemoveContainer" containerID="e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6" Oct 30 13:45:20.597295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.597272 5049 scope.go:122] "RemoveContainer" containerID="e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6" Oct 30 13:45:20.597627 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:20.597604 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6\": container with ID starting with e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6 not found: ID does not exist" containerID="e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6" Oct 30 13:45:20.597679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.597636 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6"} err="failed to get container status \"e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6\": rpc error: code = NotFound desc = could not find container \"e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6\": container with ID starting with e1595e6790775571f42df57d5750dee8342afcbb32d3372834c64823874d14a6 not found: ID does not exist" Oct 30 13:45:20.633009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.632968 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2"] Oct 30 13:45:20.653736 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.645658 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-cda0d9-748f68bc7b-lf8f2"] Oct 30 13:45:20.949278 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.949246 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:45:20.968417 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.968393 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:45:20.991700 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.991647 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e5f412-4939-44cd-937d-2bf38707178e-kube-api-access-xng7h\" (UniqueName: \"kubernetes.io/projected/91e5f412-4939-44cd-937d-2bf38707178e-kube-api-access-xng7h\") pod \"91e5f412-4939-44cd-937d-2bf38707178e\" (UID: \"91e5f412-4939-44cd-937d-2bf38707178e\") " Oct 30 13:45:20.991847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.991799 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f9f0d5-52a1-434f-acb9-fdbfe8671335-kube-api-access-86st8\" (UniqueName: \"kubernetes.io/projected/55f9f0d5-52a1-434f-acb9-fdbfe8671335-kube-api-access-86st8\") pod \"55f9f0d5-52a1-434f-acb9-fdbfe8671335\" (UID: \"55f9f0d5-52a1-434f-acb9-fdbfe8671335\") " Oct 30 13:45:20.991925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.991895 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f9f0d5-52a1-434f-acb9-fdbfe8671335-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55f9f0d5-52a1-434f-acb9-fdbfe8671335-plugins-config-volume\") pod \"55f9f0d5-52a1-434f-acb9-fdbfe8671335\" (UID: \"55f9f0d5-52a1-434f-acb9-fdbfe8671335\") " Oct 30 13:45:20.991982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.991922 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e5f412-4939-44cd-937d-2bf38707178e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e5f412-4939-44cd-937d-2bf38707178e-plugins-config-volume\") pod \"91e5f412-4939-44cd-937d-2bf38707178e\" (UID: \"91e5f412-4939-44cd-937d-2bf38707178e\") " Oct 30 13:45:20.992498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.992473 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e5f412-4939-44cd-937d-2bf38707178e-plugins-config-volume" pod "91e5f412-4939-44cd-937d-2bf38707178e" (UID: "91e5f412-4939-44cd-937d-2bf38707178e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:20.993209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.993181 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f9f0d5-52a1-434f-acb9-fdbfe8671335-plugins-config-volume" pod "55f9f0d5-52a1-434f-acb9-fdbfe8671335" (UID: "55f9f0d5-52a1-434f-acb9-fdbfe8671335"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:20.995276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.995246 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e5f412-4939-44cd-937d-2bf38707178e-kube-api-access-xng7h" pod "91e5f412-4939-44cd-937d-2bf38707178e" (UID: "91e5f412-4939-44cd-937d-2bf38707178e"). InnerVolumeSpecName "kube-api-access-xng7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:20.995437 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:20.995382 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f9f0d5-52a1-434f-acb9-fdbfe8671335-kube-api-access-86st8" pod "55f9f0d5-52a1-434f-acb9-fdbfe8671335" (UID: "55f9f0d5-52a1-434f-acb9-fdbfe8671335"). InnerVolumeSpecName "kube-api-access-86st8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:21.093069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.093033 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55f9f0d5-52a1-434f-acb9-fdbfe8671335-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:21.093069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.093069 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e5f412-4939-44cd-937d-2bf38707178e-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:21.093304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.093081 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xng7h\" (UniqueName: \"kubernetes.io/projected/91e5f412-4939-44cd-937d-2bf38707178e-kube-api-access-xng7h\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:21.093304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.093094 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86st8\" (UniqueName: \"kubernetes.io/projected/55f9f0d5-52a1-434f-acb9-fdbfe8671335-kube-api-access-86st8\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:21.335810 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.335779 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa7700f-de5f-4302-8147-edb31d44f888" path="/var/lib/kubelet/pods/3fa7700f-de5f-4302-8147-edb31d44f888/volumes" Oct 30 13:45:21.336229 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.336215 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e503802-8d23-4b8c-a357-34c464db3be3" path="/var/lib/kubelet/pods/7e503802-8d23-4b8c-a357-34c464db3be3/volumes" Oct 30 13:45:21.471885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.471844 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:21.595636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.595544 5049 generic.go:356] "Generic (PLEG): container finished" podID="91e5f412-4939-44cd-937d-2bf38707178e" containerID="21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b" exitCode=0 Oct 30 13:45:21.595636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.595596 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" Oct 30 13:45:21.595636 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.595616 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" event={"ID":"91e5f412-4939-44cd-937d-2bf38707178e","Type":"ContainerDied","Data":"21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b"} Oct 30 13:45:21.596219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.595675 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v" event={"ID":"91e5f412-4939-44cd-937d-2bf38707178e","Type":"ContainerDied","Data":"d7d89d30c1d293b74742b11719a82badd27b4aeb039798f0d123967558304bcd"} Oct 30 13:45:21.596219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.595697 5049 scope.go:122] "RemoveContainer" containerID="21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b" Oct 30 13:45:21.597160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.597134 5049 generic.go:356] "Generic (PLEG): container finished" podID="55f9f0d5-52a1-434f-acb9-fdbfe8671335" containerID="2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4" exitCode=0 Oct 30 13:45:21.597299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.597170 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" Oct 30 13:45:21.597299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.597216 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" event={"ID":"55f9f0d5-52a1-434f-acb9-fdbfe8671335","Type":"ContainerDied","Data":"2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4"} Oct 30 13:45:21.597299 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.597267 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj" event={"ID":"55f9f0d5-52a1-434f-acb9-fdbfe8671335","Type":"ContainerDied","Data":"a1c56d5df53f72350893f377824e306b4d5b1f228e4d5588ce6793a0ce7c4035"} Oct 30 13:45:21.607127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.607105 5049 scope.go:122] "RemoveContainer" containerID="21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b" Oct 30 13:45:21.607542 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:21.607520 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b\": container with ID starting with 21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b not found: ID does not exist" containerID="21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b" Oct 30 13:45:21.607668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.607552 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b"} err="failed to get container status \"21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b\": rpc error: code = NotFound desc = could not find container \"21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b\": container with ID starting with 21803591ba19ddd6521ea77887b6310bfc6dc7c1f15709a18cdad1b60cd3f76b not found: ID does not exist" Oct 30 13:45:21.607668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.607578 5049 scope.go:122] "RemoveContainer" containerID="2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4" Oct 30 13:45:21.619717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.619697 5049 scope.go:122] "RemoveContainer" containerID="2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4" Oct 30 13:45:21.620015 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:21.619995 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4\": container with ID starting with 2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4 not found: ID does not exist" containerID="2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4" Oct 30 13:45:21.620093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.620025 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4"} err="failed to get container status \"2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4\": rpc error: code = NotFound desc = could not find container \"2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4\": container with ID starting with 2d9dc9c4891292d2f0386dd8f403a0992c76e1025df16a0f0ab0fed1e283cad4 not found: ID does not exist" Oct 30 13:45:21.641456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.641432 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj"] Oct 30 13:45:21.651418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.651395 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e34fd5-858c599f95-rnqzj"] Oct 30 13:45:21.675039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.675012 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v"] Oct 30 13:45:21.680974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:21.680951 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e34fd5-54f7bbb578-x294v"] Oct 30 13:45:22.260365 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:22.254701 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:22.483069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:22.483030 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:22.510109 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:22.510073 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b"] Oct 30 13:45:22.510430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:22.510358 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" containerID="cri-o://3e5cfbb7c8ba1d78767353ead0240526f7d392de149f3141aa4af23548f980f4" gracePeriod=30 Oct 30 13:45:22.645125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:22.645086 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:22.871840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:22.871808 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:23.065653 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.065613 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:23.336582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.336540 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f9f0d5-52a1-434f-acb9-fdbfe8671335" path="/var/lib/kubelet/pods/55f9f0d5-52a1-434f-acb9-fdbfe8671335/volumes" Oct 30 13:45:23.336974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.336956 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e5f412-4939-44cd-937d-2bf38707178e" path="/var/lib/kubelet/pods/91e5f412-4939-44cd-937d-2bf38707178e/volumes" Oct 30 13:45:23.497151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.497120 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:23.620971 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.620935 5049 generic.go:356] "Generic (PLEG): container finished" podID="ede810a9-3a02-490d-9a25-d0788bea4cdf" containerID="344153ca7d7bd1782a71b1b1370fcd1ace0fe26860056380a2de00b4726f5113" exitCode=0 Oct 30 13:45:23.621160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.621002 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" event={"ID":"ede810a9-3a02-490d-9a25-d0788bea4cdf","Type":"ContainerDied","Data":"344153ca7d7bd1782a71b1b1370fcd1ace0fe26860056380a2de00b4726f5113"} Oct 30 13:45:23.751483 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.751455 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:45:23.820744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.820709 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede810a9-3a02-490d-9a25-d0788bea4cdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede810a9-3a02-490d-9a25-d0788bea4cdf-plugins-config-volume\") pod \"ede810a9-3a02-490d-9a25-d0788bea4cdf\" (UID: \"ede810a9-3a02-490d-9a25-d0788bea4cdf\") " Oct 30 13:45:23.820902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.820873 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede810a9-3a02-490d-9a25-d0788bea4cdf-kube-api-access-fbl96\" (UniqueName: \"kubernetes.io/projected/ede810a9-3a02-490d-9a25-d0788bea4cdf-kube-api-access-fbl96\") pod \"ede810a9-3a02-490d-9a25-d0788bea4cdf\" (UID: \"ede810a9-3a02-490d-9a25-d0788bea4cdf\") " Oct 30 13:45:23.821156 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.821133 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede810a9-3a02-490d-9a25-d0788bea4cdf-plugins-config-volume" pod "ede810a9-3a02-490d-9a25-d0788bea4cdf" (UID: "ede810a9-3a02-490d-9a25-d0788bea4cdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:23.821242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.821229 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede810a9-3a02-490d-9a25-d0788bea4cdf-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:23.823847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.823811 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede810a9-3a02-490d-9a25-d0788bea4cdf-kube-api-access-fbl96" pod "ede810a9-3a02-490d-9a25-d0788bea4cdf" (UID: "ede810a9-3a02-490d-9a25-d0788bea4cdf"). InnerVolumeSpecName "kube-api-access-fbl96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:23.855200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.855176 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:45:23.922304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.922212 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409baa7e-092d-4d07-9a46-e611890a8458-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409baa7e-092d-4d07-9a46-e611890a8458-plugins-config-volume\") pod \"409baa7e-092d-4d07-9a46-e611890a8458\" (UID: \"409baa7e-092d-4d07-9a46-e611890a8458\") " Oct 30 13:45:23.922492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.922452 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409baa7e-092d-4d07-9a46-e611890a8458-kube-api-access-msp8s\" (UniqueName: \"kubernetes.io/projected/409baa7e-092d-4d07-9a46-e611890a8458-kube-api-access-msp8s\") pod \"409baa7e-092d-4d07-9a46-e611890a8458\" (UID: \"409baa7e-092d-4d07-9a46-e611890a8458\") " Oct 30 13:45:23.922605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.922582 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409baa7e-092d-4d07-9a46-e611890a8458-plugins-config-volume" pod "409baa7e-092d-4d07-9a46-e611890a8458" (UID: "409baa7e-092d-4d07-9a46-e611890a8458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:23.922770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.922754 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbl96\" (UniqueName: \"kubernetes.io/projected/ede810a9-3a02-490d-9a25-d0788bea4cdf-kube-api-access-fbl96\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:23.922837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.922771 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409baa7e-092d-4d07-9a46-e611890a8458-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:23.924772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:23.924747 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409baa7e-092d-4d07-9a46-e611890a8458-kube-api-access-msp8s" pod "409baa7e-092d-4d07-9a46-e611890a8458" (UID: "409baa7e-092d-4d07-9a46-e611890a8458"). InnerVolumeSpecName "kube-api-access-msp8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:24.023240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.023200 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msp8s\" (UniqueName: \"kubernetes.io/projected/409baa7e-092d-4d07-9a46-e611890a8458-kube-api-access-msp8s\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:24.629587 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.629550 5049 generic.go:356] "Generic (PLEG): container finished" podID="409baa7e-092d-4d07-9a46-e611890a8458" containerID="2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050" exitCode=0 Oct 30 13:45:24.629754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.629601 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" Oct 30 13:45:24.629754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.629637 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" event={"ID":"409baa7e-092d-4d07-9a46-e611890a8458","Type":"ContainerDied","Data":"2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050"} Oct 30 13:45:24.629754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.629688 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj" event={"ID":"409baa7e-092d-4d07-9a46-e611890a8458","Type":"ContainerDied","Data":"5d3ffd995a03b348f35e924e64e7bfe4fac230be5f7865f5d2ebf45971f4418b"} Oct 30 13:45:24.629754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.629710 5049 scope.go:122] "RemoveContainer" containerID="2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050" Oct 30 13:45:24.631489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.631455 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" event={"ID":"ede810a9-3a02-490d-9a25-d0788bea4cdf","Type":"ContainerDied","Data":"7be712c9eca1ad7027e6897e7e3d6cc89eca570cd6e4436ffb687062a9ff1493"} Oct 30 13:45:24.631489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.631482 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2" Oct 30 13:45:24.640728 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.640711 5049 scope.go:122] "RemoveContainer" containerID="2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050" Oct 30 13:45:24.641092 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:24.641008 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050\": container with ID starting with 2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050 not found: ID does not exist" containerID="2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050" Oct 30 13:45:24.641092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.641040 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050"} err="failed to get container status \"2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050\": rpc error: code = NotFound desc = could not find container \"2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050\": container with ID starting with 2f4ef45462f2bb1b989b33bdc91ffc3bf8a922cc74a3887093f9968c01d57050 not found: ID does not exist" Oct 30 13:45:24.641092 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.641062 5049 scope.go:122] "RemoveContainer" containerID="344153ca7d7bd1782a71b1b1370fcd1ace0fe26860056380a2de00b4726f5113" Oct 30 13:45:24.699858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.699824 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2"] Oct 30 13:45:24.709841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.709813 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f07dfe-869688dc9d-r6nt2"] Oct 30 13:45:24.736874 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.736849 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj"] Oct 30 13:45:24.765389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:24.765363 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f07dfe-698c64dbcf-ln7lj"] Oct 30 13:45:25.221337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.221236 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w"] Oct 30 13:45:25.221926 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.221906 5049 cpu_manager.go:414] "containerMap: removing container" podUID="55f9f0d5-52a1-434f-acb9-fdbfe8671335" containerName="payload-pre-processing" Oct 30 13:45:25.221994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.221928 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f9f0d5-52a1-434f-acb9-fdbfe8671335" containerName="payload-pre-processing" Oct 30 13:45:25.221994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.221950 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9382dbcf-317a-47db-b952-59244215f86c" containerName="cleanup" Oct 30 13:45:25.221994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.221955 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9382dbcf-317a-47db-b952-59244215f86c" containerName="cleanup" Oct 30 13:45:25.221994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.221968 5049 cpu_manager.go:414] "containerMap: removing container" podUID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" containerName="cleanup" Oct 30 13:45:25.221994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.221973 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" containerName="cleanup" Oct 30 13:45:25.221994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.221984 5049 cpu_manager.go:414] "containerMap: removing container" podUID="fd91c2b7-2153-434b-8a1c-817eabac78b0" containerName="istio-proxy" Oct 30 13:45:25.221994 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.221991 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd91c2b7-2153-434b-8a1c-817eabac78b0" containerName="istio-proxy" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222006 5049 cpu_manager.go:414] "containerMap: removing container" podUID="7851a9ca-d13b-4a9f-ab1a-9e86e44696aa" containerName="revoke-keys" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222013 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7851a9ca-d13b-4a9f-ab1a-9e86e44696aa" containerName="revoke-keys" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222023 5049 cpu_manager.go:414] "containerMap: removing container" podUID="409baa7e-092d-4d07-9a46-e611890a8458" containerName="payload-pre-processing" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222029 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409baa7e-092d-4d07-9a46-e611890a8458" containerName="payload-pre-processing" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222044 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3fa7700f-de5f-4302-8147-edb31d44f888" containerName="payload-processing" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222051 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa7700f-de5f-4302-8147-edb31d44f888" containerName="payload-processing" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222064 5049 cpu_manager.go:414] "containerMap: removing container" podUID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" containerName="cleanup" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222071 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" containerName="cleanup" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222095 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9382dbcf-317a-47db-b952-59244215f86c" containerName="cleanup" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222102 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9382dbcf-317a-47db-b952-59244215f86c" containerName="cleanup" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222114 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ede810a9-3a02-490d-9a25-d0788bea4cdf" containerName="payload-processing" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222121 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede810a9-3a02-490d-9a25-d0788bea4cdf" containerName="payload-processing" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222132 5049 cpu_manager.go:414] "containerMap: removing container" podUID="91e5f412-4939-44cd-937d-2bf38707178e" containerName="payload-processing" Oct 30 13:45:25.222217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222140 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e5f412-4939-44cd-937d-2bf38707178e" containerName="payload-processing" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222241 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9382dbcf-317a-47db-b952-59244215f86c" containerName="cleanup" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222254 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede810a9-3a02-490d-9a25-d0788bea4cdf" containerName="payload-processing" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222263 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e5f412-4939-44cd-937d-2bf38707178e" containerName="payload-processing" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222276 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f9f0d5-52a1-434f-acb9-fdbfe8671335" containerName="payload-pre-processing" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222286 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9382dbcf-317a-47db-b952-59244215f86c" containerName="cleanup" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222296 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" containerName="cleanup" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222304 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd91c2b7-2153-434b-8a1c-817eabac78b0" containerName="istio-proxy" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222314 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9382dbcf-317a-47db-b952-59244215f86c" containerName="cleanup" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222336 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="7851a9ca-d13b-4a9f-ab1a-9e86e44696aa" containerName="revoke-keys" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222345 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="409baa7e-092d-4d07-9a46-e611890a8458" containerName="payload-pre-processing" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222353 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa7700f-de5f-4302-8147-edb31d44f888" containerName="payload-processing" Oct 30 13:45:25.222648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.222363 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fda2f4-2ef1-46c6-a85b-cfce89fe474e" containerName="cleanup" Oct 30 13:45:25.227541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.227517 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.246287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.246264 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:45:25.246963 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.246937 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w"] Oct 30 13:45:25.332468 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.332434 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11d521cb-55x4w\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.332662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.332531 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw2h6\" (UniqueName: \"kubernetes.io/projected/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-kube-api-access-cw2h6\") pod \"maas-api-revoke-sub-11d521cb-55x4w\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.332662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.332584 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11d521cb-55x4w\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.334311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.334286 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409baa7e-092d-4d07-9a46-e611890a8458" path="/var/lib/kubelet/pods/409baa7e-092d-4d07-9a46-e611890a8458/volumes" Oct 30 13:45:25.334678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.334664 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede810a9-3a02-490d-9a25-d0788bea4cdf" path="/var/lib/kubelet/pods/ede810a9-3a02-490d-9a25-d0788bea4cdf/volumes" Oct 30 13:45:25.433165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.433128 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw2h6\" (UniqueName: \"kubernetes.io/projected/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-kube-api-access-cw2h6\") pod \"maas-api-revoke-sub-11d521cb-55x4w\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.433406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.433186 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11d521cb-55x4w\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.433406 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.433237 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11d521cb-55x4w\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.434110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.434081 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11d521cb-55x4w\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.436160 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.436134 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11d521cb-55x4w\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.455403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.455373 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw2h6\" (UniqueName: \"kubernetes.io/projected/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-kube-api-access-cw2h6\") pod \"maas-api-revoke-sub-11d521cb-55x4w\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.541054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.540954 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:25.817779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.817747 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:45:25.837048 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.837017 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e86a92d-8512-45b0-8880-ef6432fcf7ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e86a92d-8512-45b0-8880-ef6432fcf7ff-plugins-config-volume\") pod \"3e86a92d-8512-45b0-8880-ef6432fcf7ff\" (UID: \"3e86a92d-8512-45b0-8880-ef6432fcf7ff\") " Oct 30 13:45:25.837210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.837143 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e86a92d-8512-45b0-8880-ef6432fcf7ff-kube-api-access-g7tzn\" (UniqueName: \"kubernetes.io/projected/3e86a92d-8512-45b0-8880-ef6432fcf7ff-kube-api-access-g7tzn\") pod \"3e86a92d-8512-45b0-8880-ef6432fcf7ff\" (UID: \"3e86a92d-8512-45b0-8880-ef6432fcf7ff\") " Oct 30 13:45:25.837449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.837420 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e86a92d-8512-45b0-8880-ef6432fcf7ff-plugins-config-volume" pod "3e86a92d-8512-45b0-8880-ef6432fcf7ff" (UID: "3e86a92d-8512-45b0-8880-ef6432fcf7ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:25.839238 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.839212 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e86a92d-8512-45b0-8880-ef6432fcf7ff-kube-api-access-g7tzn" pod "3e86a92d-8512-45b0-8880-ef6432fcf7ff" (UID: "3e86a92d-8512-45b0-8880-ef6432fcf7ff"). InnerVolumeSpecName "kube-api-access-g7tzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:25.860150 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.860120 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w"] Oct 30 13:45:25.864643 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:45:25.864613 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1e8fae_1c7e_4cca_8ba5_53fdb6740834.slice/crio-831480575fa3deca23c737b3d05b660eaa423e9fc3fd94d9cf6649ace224a6b0 WatchSource:0}: Error finding container 831480575fa3deca23c737b3d05b660eaa423e9fc3fd94d9cf6649ace224a6b0: Status 404 returned error can't find the container with id 831480575fa3deca23c737b3d05b660eaa423e9fc3fd94d9cf6649ace224a6b0 Oct 30 13:45:25.938887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.938850 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7tzn\" (UniqueName: \"kubernetes.io/projected/3e86a92d-8512-45b0-8880-ef6432fcf7ff-kube-api-access-g7tzn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:25.938887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:25.938888 5049 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e86a92d-8512-45b0-8880-ef6432fcf7ff-plugins-config-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:26.656742 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.656707 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" event={"ID":"da1e8fae-1c7e-4cca-8ba5-53fdb6740834","Type":"ContainerStarted","Data":"5089b8aa008d9b308c62c9f8f6f3cc6adb35094f3e3f5b376cb1eea0e6be5e3c"} Oct 30 13:45:26.656927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.656767 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" event={"ID":"da1e8fae-1c7e-4cca-8ba5-53fdb6740834","Type":"ContainerStarted","Data":"831480575fa3deca23c737b3d05b660eaa423e9fc3fd94d9cf6649ace224a6b0"} Oct 30 13:45:26.658520 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.658490 5049 generic.go:356] "Generic (PLEG): container finished" podID="3e86a92d-8512-45b0-8880-ef6432fcf7ff" containerID="05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5" exitCode=0 Oct 30 13:45:26.658677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.658554 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" Oct 30 13:45:26.658677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.658578 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" event={"ID":"3e86a92d-8512-45b0-8880-ef6432fcf7ff","Type":"ContainerDied","Data":"05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5"} Oct 30 13:45:26.658677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.658630 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm" event={"ID":"3e86a92d-8512-45b0-8880-ef6432fcf7ff","Type":"ContainerDied","Data":"547ada52589864010de8203e005dd9318e2214d49c3a2b6638daae428aacfe0b"} Oct 30 13:45:26.658677 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.658647 5049 scope.go:122] "RemoveContainer" containerID="05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5" Oct 30 13:45:26.670654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.670527 5049 scope.go:122] "RemoveContainer" containerID="05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5" Oct 30 13:45:26.670920 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:26.670889 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5\": container with ID starting with 05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5 not found: ID does not exist" containerID="05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5" Oct 30 13:45:26.671073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.670930 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5"} err="failed to get container status \"05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5\": rpc error: code = NotFound desc = could not find container \"05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5\": container with ID starting with 05e16da99c328f7a2b5b68289f60e6958d1e172465a38a7f97216a79a05155f5 not found: ID does not exist" Oct 30 13:45:26.682498 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:26.682471 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:45:26.700931 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.700878 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" podStartSLOduration=1.700863215 podStartE2EDuration="1.700863215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:26.698700659 +0000 UTC m=+3233.513895083" watchObservedRunningTime="2026-10-30 13:45:26.700863215 +0000 UTC m=+3233.516058017" Oct 30 13:45:26.739147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.739112 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm"] Oct 30 13:45:26.742599 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.742571 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-52a649-8699f8bbb-mdndm"] Oct 30 13:45:26.753300 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.753273 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:26.918418 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:26.918311 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:27.022932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.022897 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8"] Oct 30 13:45:27.023171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.023144 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" containerID="cri-o://07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1" gracePeriod=30 Oct 30 13:45:27.115149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.113723 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:27.338183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.338146 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e86a92d-8512-45b0-8880-ef6432fcf7ff" path="/var/lib/kubelet/pods/3e86a92d-8512-45b0-8880-ef6432fcf7ff/volumes" Oct 30 13:45:27.365791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.365759 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:27.562221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.562195 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:27.670765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.670732 5049 generic.go:356] "Generic (PLEG): container finished" podID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerID="3e5cfbb7c8ba1d78767353ead0240526f7d392de149f3141aa4af23548f980f4" exitCode=0 Oct 30 13:45:27.670957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.670919 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" event={"ID":"e5efb124-0adc-4302-a97d-2c8f57e42a4a","Type":"ContainerDied","Data":"3e5cfbb7c8ba1d78767353ead0240526f7d392de149f3141aa4af23548f980f4"} Oct 30 13:45:27.672560 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.672528 5049 generic.go:356] "Generic (PLEG): container finished" podID="da1e8fae-1c7e-4cca-8ba5-53fdb6740834" containerID="5089b8aa008d9b308c62c9f8f6f3cc6adb35094f3e3f5b376cb1eea0e6be5e3c" exitCode=0 Oct 30 13:45:27.672959 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.672938 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" event={"ID":"da1e8fae-1c7e-4cca-8ba5-53fdb6740834","Type":"ContainerDied","Data":"5089b8aa008d9b308c62c9f8f6f3cc6adb35094f3e3f5b376cb1eea0e6be5e3c"} Oct 30 13:45:27.780879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.780849 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:45:27.857743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.857712 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-podinfo\") pod \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " Oct 30 13:45:27.857917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.857767 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-credential-socket\") pod \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " Oct 30 13:45:27.857917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.857851 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-socket\") pod \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " Oct 30 13:45:27.857917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.857887 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-certs\") pod \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " Oct 30 13:45:27.858075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.857930 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-data\") pod \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " Oct 30 13:45:27.858075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.857970 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-kube-api-access-5x45f\" (UniqueName: \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-kube-api-access-5x45f\") pod \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " Oct 30 13:45:27.858075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.858023 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istiod-ca-cert\") pod \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " Oct 30 13:45:27.858075 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.858065 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-token\" (UniqueName: \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-token\") pod \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " Oct 30 13:45:27.858295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.858108 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-envoy\") pod \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\" (UID: \"e5efb124-0adc-4302-a97d-2c8f57e42a4a\") " Oct 30 13:45:27.858295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.858169 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-credential-socket" pod "e5efb124-0adc-4302-a97d-2c8f57e42a4a" (UID: "e5efb124-0adc-4302-a97d-2c8f57e42a4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:27.858440 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.858392 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-socket" pod "e5efb124-0adc-4302-a97d-2c8f57e42a4a" (UID: "e5efb124-0adc-4302-a97d-2c8f57e42a4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:27.858497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.858438 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-data" pod "e5efb124-0adc-4302-a97d-2c8f57e42a4a" (UID: "e5efb124-0adc-4302-a97d-2c8f57e42a4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:27.858551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.858501 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.858551 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.858523 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.858848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.858822 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istiod-ca-cert" pod "e5efb124-0adc-4302-a97d-2c8f57e42a4a" (UID: "e5efb124-0adc-4302-a97d-2c8f57e42a4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:27.858957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.858930 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-certs" pod "e5efb124-0adc-4302-a97d-2c8f57e42a4a" (UID: "e5efb124-0adc-4302-a97d-2c8f57e42a4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:27.860541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.860510 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-podinfo" pod "e5efb124-0adc-4302-a97d-2c8f57e42a4a" (UID: "e5efb124-0adc-4302-a97d-2c8f57e42a4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:27.860986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.860959 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-kube-api-access-5x45f" pod "e5efb124-0adc-4302-a97d-2c8f57e42a4a" (UID: "e5efb124-0adc-4302-a97d-2c8f57e42a4a"). InnerVolumeSpecName "kube-api-access-5x45f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:27.861082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.860997 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-token" pod "e5efb124-0adc-4302-a97d-2c8f57e42a4a" (UID: "e5efb124-0adc-4302-a97d-2c8f57e42a4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:27.861082 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.861050 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-envoy" pod "e5efb124-0adc-4302-a97d-2c8f57e42a4a" (UID: "e5efb124-0adc-4302-a97d-2c8f57e42a4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:27.959869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.959778 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.959869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.959828 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.959869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.959839 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x45f\" (UniqueName: \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-kube-api-access-5x45f\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.959869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.959850 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.959869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.959862 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.959869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.959874 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.960439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:27.959882 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5efb124-0adc-4302-a97d-2c8f57e42a4a-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:28.289930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.289853 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:28.449277 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.449236 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:28.610916 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.610878 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4"] Oct 30 13:45:28.611161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.611134 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" containerID="cri-o://6079ce6bcb8476e702e7287127384a27ab56e63c8b6b4654899a11635540cecf" gracePeriod=30 Oct 30 13:45:28.622083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.622056 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:28.682577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.682548 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" Oct 30 13:45:28.682867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.682590 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b" event={"ID":"e5efb124-0adc-4302-a97d-2c8f57e42a4a","Type":"ContainerDied","Data":"35f20af8864ca6337760a60a34c57fdd1478e9a1793aef4ce573f62c259c6181"} Oct 30 13:45:28.682867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.682651 5049 scope.go:122] "RemoveContainer" containerID="3e5cfbb7c8ba1d78767353ead0240526f7d392de149f3141aa4af23548f980f4" Oct 30 13:45:28.731995 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.731959 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b"] Oct 30 13:45:28.735572 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.735543 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-8lq7b"] Oct 30 13:45:28.782905 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.782873 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:28.821985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.821957 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:28.869371 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.869115 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-kube-api-access-cw2h6\" (UniqueName: \"kubernetes.io/projected/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-kube-api-access-cw2h6\") pod \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " Oct 30 13:45:28.869746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.869592 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-xks-ca\") pod \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " Oct 30 13:45:28.869746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.869690 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-openshift-ca\") pod \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\" (UID: \"da1e8fae-1c7e-4cca-8ba5-53fdb6740834\") " Oct 30 13:45:28.870849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.870659 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-openshift-ca" pod "da1e8fae-1c7e-4cca-8ba5-53fdb6740834" (UID: "da1e8fae-1c7e-4cca-8ba5-53fdb6740834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:28.874358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.872885 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-xks-ca" pod "da1e8fae-1c7e-4cca-8ba5-53fdb6740834" (UID: "da1e8fae-1c7e-4cca-8ba5-53fdb6740834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:28.874358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.874217 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-kube-api-access-cw2h6" pod "da1e8fae-1c7e-4cca-8ba5-53fdb6740834" (UID: "da1e8fae-1c7e-4cca-8ba5-53fdb6740834"). InnerVolumeSpecName "kube-api-access-cw2h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:28.971101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.971056 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:28.971101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.971094 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:28.971101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:28.971104 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw2h6\" (UniqueName: \"kubernetes.io/projected/da1e8fae-1c7e-4cca-8ba5-53fdb6740834-kube-api-access-cw2h6\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:29.334554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:29.334519 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" path="/var/lib/kubelet/pods/e5efb124-0adc-4302-a97d-2c8f57e42a4a/volumes" Oct 30 13:45:29.693166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:29.693087 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" Oct 30 13:45:29.693166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:29.693101 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w" event={"ID":"da1e8fae-1c7e-4cca-8ba5-53fdb6740834","Type":"ContainerDied","Data":"831480575fa3deca23c737b3d05b660eaa423e9fc3fd94d9cf6649ace224a6b0"} Oct 30 13:45:29.693166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:29.693150 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="831480575fa3deca23c737b3d05b660eaa423e9fc3fd94d9cf6649ace224a6b0" Oct 30 13:45:30.608526 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:30.608493 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:31.185076 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:31.185033 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:31.358793 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:31.358765 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:31.503057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:31.502971 5049 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:45:31.503057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:31.503029 5049 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:45:31.951895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:31.951860 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:32.294430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.294406 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:45:32.405632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.405598 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-data\") pod \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " Oct 30 13:45:32.405823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.405638 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-envoy\") pod \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " Oct 30 13:45:32.405823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.405683 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-token\" (UniqueName: \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-token\") pod \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " Oct 30 13:45:32.405823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.405710 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-certs\") pod \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " Oct 30 13:45:32.405823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.405749 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-socket\") pod \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " Oct 30 13:45:32.405823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.405783 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-kube-api-access-gp7m7\" (UniqueName: \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-kube-api-access-gp7m7\") pod \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " Oct 30 13:45:32.405823 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.405818 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-credential-socket\") pod \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " Oct 30 13:45:32.406115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.405848 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istiod-ca-cert\") pod \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " Oct 30 13:45:32.406115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.405905 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-podinfo\") pod \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\" (UID: \"c7ad6423-a145-4a83-be4f-12fad5cb0a51\") " Oct 30 13:45:32.406115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.405950 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-data" pod "c7ad6423-a145-4a83-be4f-12fad5cb0a51" (UID: "c7ad6423-a145-4a83-be4f-12fad5cb0a51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.406294 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.406191 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-socket" pod "c7ad6423-a145-4a83-be4f-12fad5cb0a51" (UID: "c7ad6423-a145-4a83-be4f-12fad5cb0a51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.406294 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.406254 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-credential-socket" pod "c7ad6423-a145-4a83-be4f-12fad5cb0a51" (UID: "c7ad6423-a145-4a83-be4f-12fad5cb0a51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.406518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.406363 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-certs" pod "c7ad6423-a145-4a83-be4f-12fad5cb0a51" (UID: "c7ad6423-a145-4a83-be4f-12fad5cb0a51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.406518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.406447 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istiod-ca-cert" pod "c7ad6423-a145-4a83-be4f-12fad5cb0a51" (UID: "c7ad6423-a145-4a83-be4f-12fad5cb0a51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:32.407712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.407690 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.407821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.407717 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.407821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.407738 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.407821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.407751 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.407821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.407765 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.408509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.408475 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-token" pod "c7ad6423-a145-4a83-be4f-12fad5cb0a51" (UID: "c7ad6423-a145-4a83-be4f-12fad5cb0a51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:32.408614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.408552 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-podinfo" pod "c7ad6423-a145-4a83-be4f-12fad5cb0a51" (UID: "c7ad6423-a145-4a83-be4f-12fad5cb0a51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:32.408656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.408605 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-envoy" pod "c7ad6423-a145-4a83-be4f-12fad5cb0a51" (UID: "c7ad6423-a145-4a83-be4f-12fad5cb0a51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.408656 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.408628 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-kube-api-access-gp7m7" pod "c7ad6423-a145-4a83-be4f-12fad5cb0a51" (UID: "c7ad6423-a145-4a83-be4f-12fad5cb0a51"). InnerVolumeSpecName "kube-api-access-gp7m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:32.508688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.508657 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.508688 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.508684 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.508894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.508697 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp7m7\" (UniqueName: \"kubernetes.io/projected/c7ad6423-a145-4a83-be4f-12fad5cb0a51-kube-api-access-gp7m7\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.508894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.508710 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ad6423-a145-4a83-be4f-12fad5cb0a51-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.722537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.722497 5049 generic.go:356] "Generic (PLEG): container finished" podID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerID="07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1" exitCode=0 Oct 30 13:45:32.722765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.722646 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" event={"ID":"c7ad6423-a145-4a83-be4f-12fad5cb0a51","Type":"ContainerDied","Data":"07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1"} Oct 30 13:45:32.722765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.722695 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" event={"ID":"c7ad6423-a145-4a83-be4f-12fad5cb0a51","Type":"ContainerDied","Data":"3c115e23f5319f14efc9a1ce02f78e4211e7d60757e6d9ec7a84b43ead9b1680"} Oct 30 13:45:32.722765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.722718 5049 scope.go:122] "RemoveContainer" containerID="07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1" Oct 30 13:45:32.722947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.722844 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8" Oct 30 13:45:32.742449 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.742426 5049 scope.go:122] "RemoveContainer" containerID="1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442" Oct 30 13:45:32.759914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.759887 5049 scope.go:122] "RemoveContainer" containerID="07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1" Oct 30 13:45:32.760342 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:32.760296 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1\": container with ID starting with 07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1 not found: ID does not exist" containerID="07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1" Oct 30 13:45:32.760475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.760357 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1"} err="failed to get container status \"07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1\": rpc error: code = NotFound desc = could not find container \"07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1\": container with ID starting with 07fa5d4a8d28a10752a229118a4a1331d3c240df58c4ba4407536ddc92fd96f1 not found: ID does not exist" Oct 30 13:45:32.760475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.760386 5049 scope.go:122] "RemoveContainer" containerID="1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442" Oct 30 13:45:32.760837 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:32.760802 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442\": container with ID starting with 1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442 not found: ID does not exist" containerID="1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442" Oct 30 13:45:32.760936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.760910 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442"} err="failed to get container status \"1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442\": rpc error: code = NotFound desc = could not find container \"1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442\": container with ID starting with 1e13582defa29bd45e4890141afff9854f8781197a51bd0ab34b2b879319a442 not found: ID does not exist" Oct 30 13:45:32.773145 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.773121 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8"] Oct 30 13:45:32.778546 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.778522 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-5qkz8"] Oct 30 13:45:32.840389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.840359 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:32.993382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:32.993283 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:33.334536 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.334506 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" path="/var/lib/kubelet/pods/c7ad6423-a145-4a83-be4f-12fad5cb0a51/volumes" Oct 30 13:45:33.565699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.565668 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" Oct 30 13:45:33.623399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.622593 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c311b4f0-1429-4b35-ad65-250c5404b0c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c311b4f0-1429-4b35-ad65-250c5404b0c8-maas-api-service-ca\") pod \"c311b4f0-1429-4b35-ad65-250c5404b0c8\" (UID: \"c311b4f0-1429-4b35-ad65-250c5404b0c8\") " Oct 30 13:45:33.623399 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.622750 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c311b4f0-1429-4b35-ad65-250c5404b0c8-kube-api-access-vlhwq\" (UniqueName: \"kubernetes.io/projected/c311b4f0-1429-4b35-ad65-250c5404b0c8-kube-api-access-vlhwq\") pod \"c311b4f0-1429-4b35-ad65-250c5404b0c8\" (UID: \"c311b4f0-1429-4b35-ad65-250c5404b0c8\") " Oct 30 13:45:33.624217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.624165 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c311b4f0-1429-4b35-ad65-250c5404b0c8-maas-api-service-ca" pod "c311b4f0-1429-4b35-ad65-250c5404b0c8" (UID: "c311b4f0-1429-4b35-ad65-250c5404b0c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:33.626242 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.626201 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c311b4f0-1429-4b35-ad65-250c5404b0c8-kube-api-access-vlhwq" pod "c311b4f0-1429-4b35-ad65-250c5404b0c8" (UID: "c311b4f0-1429-4b35-ad65-250c5404b0c8"). InnerVolumeSpecName "kube-api-access-vlhwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:33.723562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.723533 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlhwq\" (UniqueName: \"kubernetes.io/projected/c311b4f0-1429-4b35-ad65-250c5404b0c8-kube-api-access-vlhwq\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:33.723562 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.723559 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c311b4f0-1429-4b35-ad65-250c5404b0c8-maas-api-service-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:33.734774 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.734745 5049 generic.go:356] "Generic (PLEG): container finished" podID="f745de12-8e89-4082-9789-262034268d63" containerID="6079ce6bcb8476e702e7287127384a27ab56e63c8b6b4654899a11635540cecf" exitCode=0 Oct 30 13:45:33.734886 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.734814 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" event={"ID":"f745de12-8e89-4082-9789-262034268d63","Type":"ContainerDied","Data":"6079ce6bcb8476e702e7287127384a27ab56e63c8b6b4654899a11635540cecf"} Oct 30 13:45:33.736752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.736724 5049 generic.go:356] "Generic (PLEG): container finished" podID="c311b4f0-1429-4b35-ad65-250c5404b0c8" containerID="04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d" exitCode=137 Oct 30 13:45:33.736876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.736771 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" Oct 30 13:45:33.736876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.736776 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" event={"ID":"c311b4f0-1429-4b35-ad65-250c5404b0c8","Type":"ContainerDied","Data":"04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d"} Oct 30 13:45:33.736876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.736826 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl" event={"ID":"c311b4f0-1429-4b35-ad65-250c5404b0c8","Type":"ContainerDied","Data":"031f53290cf5f869d459c957be986aa94b36bc9aec58100c98f75129a6c8197e"} Oct 30 13:45:33.736876 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.736863 5049 scope.go:122] "RemoveContainer" containerID="04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d" Oct 30 13:45:33.749431 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.749402 5049 scope.go:122] "RemoveContainer" containerID="c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545" Oct 30 13:45:33.758863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.758839 5049 scope.go:122] "RemoveContainer" containerID="04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d" Oct 30 13:45:33.759151 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:33.759132 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d\": container with ID starting with 04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d not found: ID does not exist" containerID="04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d" Oct 30 13:45:33.759195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.759161 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d"} err="failed to get container status \"04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d\": rpc error: code = NotFound desc = could not find container \"04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d\": container with ID starting with 04ee5435dd5e642d12d664b4069ed2773586bc68ee9a717fb89e605daee2523d not found: ID does not exist" Oct 30 13:45:33.759195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.759183 5049 scope.go:122] "RemoveContainer" containerID="c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545" Oct 30 13:45:33.759424 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:33.759409 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545\": container with ID starting with c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545 not found: ID does not exist" containerID="c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545" Oct 30 13:45:33.759473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.759428 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545"} err="failed to get container status \"c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545\": rpc error: code = NotFound desc = could not find container \"c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545\": container with ID starting with c0fef74432e56abe4f7fcdf48eb4c87a70aa2844051b4b95e40e4ca86c26f545 not found: ID does not exist" Oct 30 13:45:33.782877 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.782843 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl"] Oct 30 13:45:33.787710 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.787653 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889465-sjjvl"] Oct 30 13:45:33.849612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.849579 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:33.877018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.876955 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:45:33.925937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.925892 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-data\") pod \"f745de12-8e89-4082-9789-262034268d63\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " Oct 30 13:45:33.926120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.925967 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-envoy\") pod \"f745de12-8e89-4082-9789-262034268d63\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " Oct 30 13:45:33.926120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.925999 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-kube-api-access-h8nbr\" (UniqueName: \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-kube-api-access-h8nbr\") pod \"f745de12-8e89-4082-9789-262034268d63\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " Oct 30 13:45:33.926120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926043 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-istio-token\" (UniqueName: \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-istio-token\") pod \"f745de12-8e89-4082-9789-262034268d63\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " Oct 30 13:45:33.926120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926073 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-socket\") pod \"f745de12-8e89-4082-9789-262034268d63\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " Oct 30 13:45:33.926120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926117 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f745de12-8e89-4082-9789-262034268d63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745de12-8e89-4082-9789-262034268d63-istio-podinfo\") pod \"f745de12-8e89-4082-9789-262034268d63\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " Oct 30 13:45:33.926412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926147 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f745de12-8e89-4082-9789-262034268d63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745de12-8e89-4082-9789-262034268d63-istiod-ca-cert\") pod \"f745de12-8e89-4082-9789-262034268d63\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " Oct 30 13:45:33.926412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926208 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-data" pod "f745de12-8e89-4082-9789-262034268d63" (UID: "f745de12-8e89-4082-9789-262034268d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:33.926412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926238 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-certs\") pod \"f745de12-8e89-4082-9789-262034268d63\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " Oct 30 13:45:33.926412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926268 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-credential-socket\") pod \"f745de12-8e89-4082-9789-262034268d63\" (UID: \"f745de12-8e89-4082-9789-262034268d63\") " Oct 30 13:45:33.926681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926453 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-socket" pod "f745de12-8e89-4082-9789-262034268d63" (UID: "f745de12-8e89-4082-9789-262034268d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:33.926998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926952 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-credential-socket" pod "f745de12-8e89-4082-9789-262034268d63" (UID: "f745de12-8e89-4082-9789-262034268d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:33.926998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926971 5049 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-data\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:33.926998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926997 5049 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:33.927271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.926967 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-certs" pod "f745de12-8e89-4082-9789-262034268d63" (UID: "f745de12-8e89-4082-9789-262034268d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:33.927271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.927099 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f745de12-8e89-4082-9789-262034268d63-istiod-ca-cert" pod "f745de12-8e89-4082-9789-262034268d63" (UID: "f745de12-8e89-4082-9789-262034268d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:33.929070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.929038 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-envoy" pod "f745de12-8e89-4082-9789-262034268d63" (UID: "f745de12-8e89-4082-9789-262034268d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:33.929199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.929160 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-kube-api-access-h8nbr" pod "f745de12-8e89-4082-9789-262034268d63" (UID: "f745de12-8e89-4082-9789-262034268d63"). InnerVolumeSpecName "kube-api-access-h8nbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:33.929368 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.929316 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f745de12-8e89-4082-9789-262034268d63-istio-podinfo" pod "f745de12-8e89-4082-9789-262034268d63" (UID: "f745de12-8e89-4082-9789-262034268d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:33.930374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:33.930346 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-istio-token" pod "f745de12-8e89-4082-9789-262034268d63" (UID: "f745de12-8e89-4082-9789-262034268d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:34.027716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.027673 5049 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-istio-envoy\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:34.027716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.027704 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8nbr\" (UniqueName: \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-kube-api-access-h8nbr\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:34.027716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.027715 5049 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745de12-8e89-4082-9789-262034268d63-istio-token\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:34.027716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.027723 5049 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745de12-8e89-4082-9789-262034268d63-istio-podinfo\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:34.027716 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.027731 5049 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745de12-8e89-4082-9789-262034268d63-istiod-ca-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:34.028280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.027739 5049 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-workload-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:34.028280 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.027749 5049 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745de12-8e89-4082-9789-262034268d63-credential-socket\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:45:34.746220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.746142 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" Oct 30 13:45:34.746220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.746166 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4" event={"ID":"f745de12-8e89-4082-9789-262034268d63","Type":"ContainerDied","Data":"450719cdfee8a6df5dccb7be8d27407ac6e04a5deaa22ab44e80c0b3b0a2c6a8"} Oct 30 13:45:34.746469 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.746230 5049 scope.go:122] "RemoveContainer" containerID="6079ce6bcb8476e702e7287127384a27ab56e63c8b6b4654899a11635540cecf" Oct 30 13:45:34.767910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.767879 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:34.803427 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.803393 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4"] Oct 30 13:45:34.813798 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:34.813766 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-fqfq4"] Oct 30 13:45:35.335341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:35.335278 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c311b4f0-1429-4b35-ad65-250c5404b0c8" path="/var/lib/kubelet/pods/c311b4f0-1429-4b35-ad65-250c5404b0c8/volumes" Oct 30 13:45:35.335937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:35.335918 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f745de12-8e89-4082-9789-262034268d63" path="/var/lib/kubelet/pods/f745de12-8e89-4082-9789-262034268d63/volumes" Oct 30 13:45:38.199943 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:38.199909 5049 scope.go:122] "RemoveContainer" containerID="0d9530422e3475df369b11a6cc7eec5071f83e643d712b8a27a330f0d8d5e943" Oct 30 13:45:38.209648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:38.209625 5049 scope.go:122] "RemoveContainer" containerID="627f8df0fee96d17b62771ad2a0a55d914316f29d8376af8844dbf61f2cdb8d7" Oct 30 13:45:38.219064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:38.219044 5049 scope.go:122] "RemoveContainer" containerID="92fb3846104009974a50456818e00bd02b6699409c6a197178b7571a4ce667a7" Oct 30 13:45:39.887553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:39.887524 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:40.051516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:40.051481 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:40.381753 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:40.381721 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:40.942022 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:40.941983 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:41.042878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:41.042840 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj"] Oct 30 13:45:41.052374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:41.052348 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee7d65-vzdrj"] Oct 30 13:45:41.152037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:41.151984 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:41.334182 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:41.334149 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7851a9ca-d13b-4a9f-ab1a-9e86e44696aa" path="/var/lib/kubelet/pods/7851a9ca-d13b-4a9f-ab1a-9e86e44696aa/volumes" Oct 30 13:45:41.630654 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:41.630568 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:48.086093 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:48.086059 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:48.261289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:48.261248 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:48.548000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:48.547967 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:54.661985 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:54.661948 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:54.977960 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:54.977879 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:55.215013 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.214963 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:55.429106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.429076 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:55.576601 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.576563 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b844f68b7-crsvk"] Oct 30 13:45:55.577175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577157 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3e86a92d-8512-45b0-8880-ef6432fcf7ff" containerName="payload-processing" Oct 30 13:45:55.577237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577178 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e86a92d-8512-45b0-8880-ef6432fcf7ff" containerName="payload-processing" Oct 30 13:45:55.577237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577202 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9382dbcf-317a-47db-b952-59244215f86c" containerName="cleanup" Oct 30 13:45:55.577237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577210 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9382dbcf-317a-47db-b952-59244215f86c" containerName="cleanup" Oct 30 13:45:55.577237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577233 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577242 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577251 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c311b4f0-1429-4b35-ad65-250c5404b0c8" containerName="cleanup" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577258 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c311b4f0-1429-4b35-ad65-250c5404b0c8" containerName="cleanup" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577275 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577283 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577301 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577309 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577334 5049 cpu_manager.go:414] "containerMap: removing container" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577344 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577367 5049 cpu_manager.go:414] "containerMap: removing container" podUID="c311b4f0-1429-4b35-ad65-250c5404b0c8" containerName="cleanup" Oct 30 13:45:55.577392 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577375 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c311b4f0-1429-4b35-ad65-250c5404b0c8" containerName="cleanup" Oct 30 13:45:55.577765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577397 5049 cpu_manager.go:414] "containerMap: removing container" podUID="da1e8fae-1c7e-4cca-8ba5-53fdb6740834" containerName="revoke-keys" Oct 30 13:45:55.577765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577405 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1e8fae-1c7e-4cca-8ba5-53fdb6740834" containerName="revoke-keys" Oct 30 13:45:55.577765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577502 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" Oct 30 13:45:55.577765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577512 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5efb124-0adc-4302-a97d-2c8f57e42a4a" containerName="istio-proxy" Oct 30 13:45:55.577765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577519 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c311b4f0-1429-4b35-ad65-250c5404b0c8" containerName="cleanup" Oct 30 13:45:55.577765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577527 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1e8fae-1c7e-4cca-8ba5-53fdb6740834" containerName="revoke-keys" Oct 30 13:45:55.577765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577532 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e86a92d-8512-45b0-8880-ef6432fcf7ff" containerName="payload-processing" Oct 30 13:45:55.577765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577539 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f745de12-8e89-4082-9789-262034268d63" containerName="istio-proxy" Oct 30 13:45:55.577765 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.577545 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c311b4f0-1429-4b35-ad65-250c5404b0c8" containerName="cleanup" Oct 30 13:45:55.588068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.588038 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:55.613485 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.613431 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 13:45:55.613866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.613842 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h5q88\"" Oct 30 13:45:55.613866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.613842 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 13:45:55.616251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.616221 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b844f68b7-crsvk"] Oct 30 13:45:55.644616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.644582 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 13:45:55.729605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.729521 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xh87\" (UniqueName: \"kubernetes.io/projected/4add8cd3-6156-4323-a224-cdcd15655f6f-kube-api-access-5xh87\") pod \"maas-controller-b844f68b7-crsvk\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:55.729605 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.729575 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-metrics-certs\") pod \"maas-controller-b844f68b7-crsvk\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:55.730043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.729718 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-webhook-cert\") pod \"maas-controller-b844f68b7-crsvk\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:55.830276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.830238 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-metrics-certs\") pod \"maas-controller-b844f68b7-crsvk\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:55.830478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.830400 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-webhook-cert\") pod \"maas-controller-b844f68b7-crsvk\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:55.830478 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.830437 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xh87\" (UniqueName: \"kubernetes.io/projected/4add8cd3-6156-4323-a224-cdcd15655f6f-kube-api-access-5xh87\") pod \"maas-controller-b844f68b7-crsvk\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:55.832789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.832763 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-metrics-certs\") pod \"maas-controller-b844f68b7-crsvk\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:55.832912 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.832799 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-webhook-cert\") pod \"maas-controller-b844f68b7-crsvk\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:55.852407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.852381 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xh87\" (UniqueName: \"kubernetes.io/projected/4add8cd3-6156-4323-a224-cdcd15655f6f-kube-api-access-5xh87\") pod \"maas-controller-b844f68b7-crsvk\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:55.902044 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:55.902003 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:56.211921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:56.211883 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b844f68b7-crsvk"] Oct 30 13:45:56.216700 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:45:56.216671 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4add8cd3_6156_4323_a224_cdcd15655f6f.slice/crio-e3ad4000953ed2d6c56e44cff47f1c5716e12c1100dc1cef2c35dd61095ec432 WatchSource:0}: Error finding container e3ad4000953ed2d6c56e44cff47f1c5716e12c1100dc1cef2c35dd61095ec432: Status 404 returned error can't find the container with id e3ad4000953ed2d6c56e44cff47f1c5716e12c1100dc1cef2c35dd61095ec432 Oct 30 13:45:56.678304 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:45:56.678265 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:45:56.708455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:56.708425 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:56.893984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:56.893947 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b844f68b7-crsvk" event={"ID":"4add8cd3-6156-4323-a224-cdcd15655f6f","Type":"ContainerStarted","Data":"e3ad4000953ed2d6c56e44cff47f1c5716e12c1100dc1cef2c35dd61095ec432"} Oct 30 13:45:57.901834 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:57.901787 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b844f68b7-crsvk" event={"ID":"4add8cd3-6156-4323-a224-cdcd15655f6f","Type":"ContainerStarted","Data":"5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908"} Oct 30 13:45:57.902284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:57.901857 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:45:57.946210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:45:57.946155 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b844f68b7-crsvk" podStartSLOduration=1.366749303 podStartE2EDuration="2.946141953s" totalImagesPullingTime="1.57939265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:57.944136995 +0000 UTC m=+3264.759331418" watchObservedRunningTime="2026-10-30 13:45:57.946141953 +0000 UTC m=+3264.761336406" Oct 30 13:46:02.311391 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:02.311358 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:08.910455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:08.910423 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:46:10.336201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.336159 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq"] Oct 30 13:46:10.336785 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.336761 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ad6423-a145-4a83-be4f-12fad5cb0a51" containerName="istio-proxy" Oct 30 13:46:10.342489 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.342467 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.358563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.358538 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 13:46:10.358752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.358727 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bwksz\"" Oct 30 13:46:10.358828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.358750 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb02668-044a-472c-be76-77c2cf53c3f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.358828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.358782 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.358828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.358799 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.359006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.358886 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb02668-044a-472c-be76-77c2cf53c3f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.359006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.358915 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb02668-044a-472c-be76-77c2cf53c3f9-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.359006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.358953 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.359006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.358976 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.359193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.359092 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.359193 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.359117 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctwvn\" (UniqueName: \"kubernetes.io/projected/deb02668-044a-472c-be76-77c2cf53c3f9-kube-api-access-ctwvn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.371707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.371683 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq"] Oct 30 13:46:10.460011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.459972 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460011 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460016 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctwvn\" (UniqueName: \"kubernetes.io/projected/deb02668-044a-472c-be76-77c2cf53c3f9-kube-api-access-ctwvn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460088 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb02668-044a-472c-be76-77c2cf53c3f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460168 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460200 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460227 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb02668-044a-472c-be76-77c2cf53c3f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460290 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460258 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb02668-044a-472c-be76-77c2cf53c3f9-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460355 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460381 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460629 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460471 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460818 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460758 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460821 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460891 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb02668-044a-472c-be76-77c2cf53c3f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.460947 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.460897 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.462571 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.462550 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb02668-044a-472c-be76-77c2cf53c3f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.462831 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.462812 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb02668-044a-472c-be76-77c2cf53c3f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.481906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.481878 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb02668-044a-472c-be76-77c2cf53c3f9-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.482066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.481914 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctwvn\" (UniqueName: \"kubernetes.io/projected/deb02668-044a-472c-be76-77c2cf53c3f9-kube-api-access-ctwvn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-b8pzq\" (UID: \"deb02668-044a-472c-be76-77c2cf53c3f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.655456 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.655367 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:10.977511 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.977432 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq"] Oct 30 13:46:10.983272 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:46:10.983240 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb02668_044a_472c_be76_77c2cf53c3f9.slice/crio-e045dec1029de349aa0e097b1162338eeb24ba7e9d6595c9201b42c66f0595d9 WatchSource:0}: Error finding container e045dec1029de349aa0e097b1162338eeb24ba7e9d6595c9201b42c66f0595d9: Status 404 returned error can't find the container with id e045dec1029de349aa0e097b1162338eeb24ba7e9d6595c9201b42c66f0595d9 Oct 30 13:46:10.985263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.985234 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:46:10.985357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.985301 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:46:10.985357 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:10.985342 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:46:11.994999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:11.994963 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" event={"ID":"deb02668-044a-472c-be76-77c2cf53c3f9","Type":"ContainerStarted","Data":"5e018e32374e41c3e605abfa477f03f65bbe356d10ff29a5bb869ed1d4cccf96"} Oct 30 13:46:11.995419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:11.995016 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" event={"ID":"deb02668-044a-472c-be76-77c2cf53c3f9","Type":"ContainerStarted","Data":"e045dec1029de349aa0e097b1162338eeb24ba7e9d6595c9201b42c66f0595d9"} Oct 30 13:46:12.656084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:12.656040 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:12.657582 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:12.657555 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-b8pzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" start-of-body= Oct 30 13:46:12.657699 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:12.657607 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" podUID="deb02668-044a-472c-be76-77c2cf53c3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" Oct 30 13:46:13.656655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:13.656611 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-b8pzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" start-of-body= Oct 30 13:46:13.657066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:13.656674 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" podUID="deb02668-044a-472c-be76-77c2cf53c3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" Oct 30 13:46:14.656255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:14.656163 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-b8pzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" start-of-body= Oct 30 13:46:14.656255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:14.656222 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" podUID="deb02668-044a-472c-be76-77c2cf53c3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" Oct 30 13:46:15.655998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:15.655963 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-b8pzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" start-of-body= Oct 30 13:46:15.656517 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:15.656023 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" podUID="deb02668-044a-472c-be76-77c2cf53c3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" Oct 30 13:46:16.656177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:16.656125 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-b8pzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" start-of-body= Oct 30 13:46:16.656691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:16.656202 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" podUID="deb02668-044a-472c-be76-77c2cf53c3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" Oct 30 13:46:17.656177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:17.656137 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-b8pzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" start-of-body= Oct 30 13:46:17.656420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:17.656204 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" podUID="deb02668-044a-472c-be76-77c2cf53c3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.112:15021/healthz/ready\": dial tcp 10.132.1.112:15021: connect: connection refused" Oct 30 13:46:19.361487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:19.361451 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:19.362130 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:19.362106 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:19.363029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:19.363007 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" Oct 30 13:46:19.425255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:19.425186 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-b8pzq" podStartSLOduration=9.425166032 podStartE2EDuration="9.425166032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:12.091473991 +0000 UTC m=+3278.906668413" watchObservedRunningTime="2026-10-30 13:46:19.425166032 +0000 UTC m=+3286.240360456" Oct 30 13:46:25.393748 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.393710 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64"] Oct 30 13:46:25.398275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.398253 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.422721 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.422684 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64"] Oct 30 13:46:25.542117 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.542076 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.542352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.542131 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e381cf09-b412-436c-a024-26f22743c7c6-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.542352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.542237 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.542352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.542273 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e381cf09-b412-436c-a024-26f22743c7c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.542352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.542294 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.542603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.542418 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.542603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.542451 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rqr\" (UniqueName: \"kubernetes.io/projected/e381cf09-b412-436c-a024-26f22743c7c6-kube-api-access-f9rqr\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.542603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.542499 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e381cf09-b412-436c-a024-26f22743c7c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.542603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.542562 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.643306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643269 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.643306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643311 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e381cf09-b412-436c-a024-26f22743c7c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.643591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643359 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.643591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643385 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.643591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643424 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rqr\" (UniqueName: \"kubernetes.io/projected/e381cf09-b412-436c-a024-26f22743c7c6-kube-api-access-f9rqr\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.643591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643449 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e381cf09-b412-436c-a024-26f22743c7c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.643591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643490 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.643591 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643541 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.644073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643598 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e381cf09-b412-436c-a024-26f22743c7c6-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.644073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643857 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.644073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.643882 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.644212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.644074 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.644212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.644186 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.644309 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.644284 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e381cf09-b412-436c-a024-26f22743c7c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.645828 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.645805 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e381cf09-b412-436c-a024-26f22743c7c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.645925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.645898 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e381cf09-b412-436c-a024-26f22743c7c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.665282 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.665253 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e381cf09-b412-436c-a024-26f22743c7c6-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.665402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.665305 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rqr\" (UniqueName: \"kubernetes.io/projected/e381cf09-b412-436c-a024-26f22743c7c6-kube-api-access-f9rqr\") pod \"maas-default-gateway-openshift-default-64d4cc998c-5tb64\" (UID: \"e381cf09-b412-436c-a024-26f22743c7c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:25.712000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:25.711968 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:26.035965 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:26.035883 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64"] Oct 30 13:46:26.039422 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:46:26.039392 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode381cf09_b412_436c_a024_26f22743c7c6.slice/crio-e06f0f52fb7985b51ce2aec3fd5bd58fcd0b72a1d9e22e23e6d59563cba188d9 WatchSource:0}: Error finding container e06f0f52fb7985b51ce2aec3fd5bd58fcd0b72a1d9e22e23e6d59563cba188d9: Status 404 returned error can't find the container with id e06f0f52fb7985b51ce2aec3fd5bd58fcd0b72a1d9e22e23e6d59563cba188d9 Oct 30 13:46:26.041866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:26.041831 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:46:26.041974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:26.041897 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:46:26.041974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:26.041930 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:46:26.135506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:26.135465 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" event={"ID":"e381cf09-b412-436c-a024-26f22743c7c6","Type":"ContainerStarted","Data":"ddb788532a46241baa76eef826258c952a6b2dcf29f14bc6c465f73b7f09eb72"} Oct 30 13:46:26.135726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:26.135531 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" event={"ID":"e381cf09-b412-436c-a024-26f22743c7c6","Type":"ContainerStarted","Data":"e06f0f52fb7985b51ce2aec3fd5bd58fcd0b72a1d9e22e23e6d59563cba188d9"} Oct 30 13:46:26.177819 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:26.177759 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" podStartSLOduration=1.177741367 podStartE2EDuration="1.177741367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:26.176710172 +0000 UTC m=+3292.991904598" watchObservedRunningTime="2026-10-30 13:46:26.177741367 +0000 UTC m=+3292.992935791" Oct 30 13:46:26.676841 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:46:26.676808 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:46:26.719346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:26.712705 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:27.730516 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:27.730474 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-5tb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body= Oct 30 13:46:27.730939 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:27.730546 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" podUID="e381cf09-b412-436c-a024-26f22743c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" Oct 30 13:46:28.719553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:28.716711 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-5tb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body= Oct 30 13:46:28.719553 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:28.716785 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" podUID="e381cf09-b412-436c-a024-26f22743c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" Oct 30 13:46:29.716191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:29.715728 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-5tb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body= Oct 30 13:46:29.716191 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:29.715793 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" podUID="e381cf09-b412-436c-a024-26f22743c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" Oct 30 13:46:30.169183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.169150 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:30.271251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.271215 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b844f68b7-crsvk"] Oct 30 13:46:30.271731 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.271666 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-b844f68b7-crsvk" podUID="4add8cd3-6156-4323-a224-cdcd15655f6f" containerName="manager" containerID="cri-o://5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908" gracePeriod=30 Oct 30 13:46:30.564005 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.563977 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:46:30.698901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.698841 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-webhook-cert\") pod \"4add8cd3-6156-4323-a224-cdcd15655f6f\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " Oct 30 13:46:30.699403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.699379 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-metrics-certs\") pod \"4add8cd3-6156-4323-a224-cdcd15655f6f\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " Oct 30 13:46:30.699578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.699555 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4add8cd3-6156-4323-a224-cdcd15655f6f-kube-api-access-5xh87\" (UniqueName: \"kubernetes.io/projected/4add8cd3-6156-4323-a224-cdcd15655f6f-kube-api-access-5xh87\") pod \"4add8cd3-6156-4323-a224-cdcd15655f6f\" (UID: \"4add8cd3-6156-4323-a224-cdcd15655f6f\") " Oct 30 13:46:30.701230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.701197 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-webhook-cert" pod "4add8cd3-6156-4323-a224-cdcd15655f6f" (UID: "4add8cd3-6156-4323-a224-cdcd15655f6f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:46:30.701452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.701435 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-metrics-certs" pod "4add8cd3-6156-4323-a224-cdcd15655f6f" (UID: "4add8cd3-6156-4323-a224-cdcd15655f6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:46:30.702764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.702707 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4add8cd3-6156-4323-a224-cdcd15655f6f-kube-api-access-5xh87" pod "4add8cd3-6156-4323-a224-cdcd15655f6f" (UID: "4add8cd3-6156-4323-a224-cdcd15655f6f"). InnerVolumeSpecName "kube-api-access-5xh87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:46:30.731573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.725694 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-5tb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body= Oct 30 13:46:30.731573 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.725766 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" podUID="e381cf09-b412-436c-a024-26f22743c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" Oct 30 13:46:30.762645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.762615 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:30.800631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.800602 5049 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-webhook-cert\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:46:30.800631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.800625 5049 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4add8cd3-6156-4323-a224-cdcd15655f6f-metrics-certs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:46:30.800631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:30.800635 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xh87\" (UniqueName: \"kubernetes.io/projected/4add8cd3-6156-4323-a224-cdcd15655f6f-kube-api-access-5xh87\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:46:31.169464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.169423 5049 generic.go:356] "Generic (PLEG): container finished" podID="4add8cd3-6156-4323-a224-cdcd15655f6f" containerID="5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908" exitCode=0 Oct 30 13:46:31.169672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.169490 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b844f68b7-crsvk" Oct 30 13:46:31.169672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.169481 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b844f68b7-crsvk" event={"ID":"4add8cd3-6156-4323-a224-cdcd15655f6f","Type":"ContainerDied","Data":"5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908"} Oct 30 13:46:31.169672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.169592 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b844f68b7-crsvk" event={"ID":"4add8cd3-6156-4323-a224-cdcd15655f6f","Type":"ContainerDied","Data":"e3ad4000953ed2d6c56e44cff47f1c5716e12c1100dc1cef2c35dd61095ec432"} Oct 30 13:46:31.169672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.169623 5049 scope.go:122] "RemoveContainer" containerID="5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908" Oct 30 13:46:31.180969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.180945 5049 scope.go:122] "RemoveContainer" containerID="5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908" Oct 30 13:46:31.181238 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:46:31.181217 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908\": container with ID starting with 5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908 not found: ID does not exist" containerID="5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908" Oct 30 13:46:31.181308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.181247 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908"} err="failed to get container status \"5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908\": rpc error: code = NotFound desc = could not find container \"5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908\": container with ID starting with 5fd14c13b9d69d46626a0c36a58c66959eb5a60c2dc7423407aed035d8d59908 not found: ID does not exist" Oct 30 13:46:31.231758 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.231718 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b844f68b7-crsvk"] Oct 30 13:46:31.236545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.236516 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b844f68b7-crsvk"] Oct 30 13:46:31.335389 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.335355 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4add8cd3-6156-4323-a224-cdcd15655f6f" path="/var/lib/kubelet/pods/4add8cd3-6156-4323-a224-cdcd15655f6f/volumes" Oct 30 13:46:31.726284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.726248 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-5tb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body= Oct 30 13:46:31.726609 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:31.726577 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" podUID="e381cf09-b412-436c-a024-26f22743c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" Oct 30 13:46:32.411015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.410977 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:32.524758 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.524716 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b844f68b7-qjmh7"] Oct 30 13:46:32.525183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.525169 5049 cpu_manager.go:414] "containerMap: removing container" podUID="4add8cd3-6156-4323-a224-cdcd15655f6f" containerName="manager" Oct 30 13:46:32.525251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.525184 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4add8cd3-6156-4323-a224-cdcd15655f6f" containerName="manager" Oct 30 13:46:32.525316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.525264 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="4add8cd3-6156-4323-a224-cdcd15655f6f" containerName="manager" Oct 30 13:46:32.530202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.530175 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:32.543614 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.543589 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 13:46:32.557288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.557257 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b844f68b7-qjmh7"] Oct 30 13:46:32.577292 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.577248 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 13:46:32.577470 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.577330 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 13:46:32.577531 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.577491 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h5q88\"" Oct 30 13:46:32.634057 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.634020 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwbp\" (UniqueName: \"kubernetes.io/projected/f7f8e4f9-5a9f-43b2-957b-16e0a8cde956-kube-api-access-pgwbp\") pod \"maas-controller-b844f68b7-qjmh7\" (UID: \"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956\") " pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:32.634247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.634069 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f8e4f9-5a9f-43b2-957b-16e0a8cde956-metrics-certs\") pod \"maas-controller-b844f68b7-qjmh7\" (UID: \"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956\") " pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:32.634247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.634162 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7f8e4f9-5a9f-43b2-957b-16e0a8cde956-webhook-cert\") pod \"maas-controller-b844f68b7-qjmh7\" (UID: \"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956\") " pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:32.712842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.712755 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-5tb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body= Oct 30 13:46:32.712842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.712823 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" podUID="e381cf09-b412-436c-a024-26f22743c7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" Oct 30 13:46:32.735421 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.735375 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f8e4f9-5a9f-43b2-957b-16e0a8cde956-metrics-certs\") pod \"maas-controller-b844f68b7-qjmh7\" (UID: \"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956\") " pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:32.735583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.735440 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7f8e4f9-5a9f-43b2-957b-16e0a8cde956-webhook-cert\") pod \"maas-controller-b844f68b7-qjmh7\" (UID: \"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956\") " pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:32.735583 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.735547 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwbp\" (UniqueName: \"kubernetes.io/projected/f7f8e4f9-5a9f-43b2-957b-16e0a8cde956-kube-api-access-pgwbp\") pod \"maas-controller-b844f68b7-qjmh7\" (UID: \"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956\") " pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:32.738074 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.738045 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7f8e4f9-5a9f-43b2-957b-16e0a8cde956-webhook-cert\") pod \"maas-controller-b844f68b7-qjmh7\" (UID: \"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956\") " pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:32.738181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.738090 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f8e4f9-5a9f-43b2-957b-16e0a8cde956-metrics-certs\") pod \"maas-controller-b844f68b7-qjmh7\" (UID: \"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956\") " pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:32.757237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.757202 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwbp\" (UniqueName: \"kubernetes.io/projected/f7f8e4f9-5a9f-43b2-957b-16e0a8cde956-kube-api-access-pgwbp\") pod \"maas-controller-b844f68b7-qjmh7\" (UID: \"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956\") " pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:32.843183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:32.843133 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:33.148895 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:33.148857 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b844f68b7-qjmh7"] Oct 30 13:46:33.153525 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:46:33.153498 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f8e4f9_5a9f_43b2_957b_16e0a8cde956.slice/crio-703c1a8abaa9af9fdf1f0187e48cc6381ae94b56265eabe53d3bdbcb3db643e0 WatchSource:0}: Error finding container 703c1a8abaa9af9fdf1f0187e48cc6381ae94b56265eabe53d3bdbcb3db643e0: Status 404 returned error can't find the container with id 703c1a8abaa9af9fdf1f0187e48cc6381ae94b56265eabe53d3bdbcb3db643e0 Oct 30 13:46:33.154870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:33.154848 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:46:33.186097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:33.186060 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b844f68b7-qjmh7" event={"ID":"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956","Type":"ContainerStarted","Data":"703c1a8abaa9af9fdf1f0187e48cc6381ae94b56265eabe53d3bdbcb3db643e0"} Oct 30 13:46:33.601413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:33.601384 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 13:46:34.063777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:34.063748 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:46:34.081032 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:34.080991 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:46:34.195162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:34.195124 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b844f68b7-qjmh7" event={"ID":"f7f8e4f9-5a9f-43b2-957b-16e0a8cde956","Type":"ContainerStarted","Data":"d303f97aa64d439f91e2d8a77eb12f2c3a3fd144137eefe76aa97eb5a50a691e"} Oct 30 13:46:34.195361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:34.195246 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:34.234521 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:34.234463 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b844f68b7-qjmh7" podStartSLOduration=1.800461178 podStartE2EDuration="2.234450692s" totalImagesPullingTime="433.989514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:34.234208352 +0000 UTC m=+3301.049402776" watchObservedRunningTime="2026-10-30 13:46:34.234450692 +0000 UTC m=+3301.049645115" Oct 30 13:46:34.414159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:34.414068 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:34.414482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:34.414453 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:34.415232 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:34.415206 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-5tb64" Oct 30 13:46:38.400121 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:38.400088 5049 scope.go:122] "RemoveContainer" containerID="6c7b915ec80165cae01306efdcea78f7b8c1ae80472711c0a50ae21d8d7135d9" Oct 30 13:46:42.028933 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:42.028898 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx"] Oct 30 13:46:42.038679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:42.038644 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56eadae9-mw8xx"] Oct 30 13:46:43.027492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:43.027447 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7"] Oct 30 13:46:43.037206 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:43.037173 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b34582ea-srgl7"] Oct 30 13:46:43.334073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:43.334039 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2c343d-fa00-4a14-bb51-5eaf1ef0d633" path="/var/lib/kubelet/pods/5f2c343d-fa00-4a14-bb51-5eaf1ef0d633/volumes" Oct 30 13:46:43.334509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:43.334491 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9c017a-fd73-46aa-8974-186145c76297" path="/var/lib/kubelet/pods/dc9c017a-fd73-46aa-8974-186145c76297/volumes" Oct 30 13:46:45.205115 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:46:45.205081 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b844f68b7-qjmh7" Oct 30 13:46:56.676963 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:46:56.676904 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:47:07.026777 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:07.026743 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5"] Oct 30 13:47:07.037052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:07.037024 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b28ef90-558g5"] Oct 30 13:47:07.334901 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:07.334868 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df406730-f0d4-4845-ada4-6d367fe059ce" path="/var/lib/kubelet/pods/df406730-f0d4-4845-ada4-6d367fe059ce/volumes" Oct 30 13:47:25.645188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.645145 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw"] Oct 30 13:47:25.649279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.649258 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.685268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.685235 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw"] Oct 30 13:47:25.736334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.736297 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da100cec-8aca-4cba-b19d-4c873acb6f2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.736509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.736356 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.736509 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.736447 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.736588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.736515 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.736625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.736601 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.736672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.736654 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.736713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.736686 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkg8\" (UniqueName: \"kubernetes.io/projected/da100cec-8aca-4cba-b19d-4c873acb6f2a-kube-api-access-mtkg8\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.736766 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.736751 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.736804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.736781 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.837667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.837630 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.837667 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.837671 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkg8\" (UniqueName: \"kubernetes.io/projected/da100cec-8aca-4cba-b19d-4c873acb6f2a-kube-api-access-mtkg8\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.837946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.837804 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.837946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.837846 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.837946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.837903 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da100cec-8aca-4cba-b19d-4c873acb6f2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.837946 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.837924 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.838199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.837950 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.838199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.837994 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.838199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.838070 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.838199 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.838137 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.838439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.838291 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.838715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.838666 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.838715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.838681 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.839163 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.839135 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da100cec-8aca-4cba-b19d-4c873acb6f2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.840178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.840152 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.840518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.840499 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.858984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.858960 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da100cec-8aca-4cba-b19d-4c873acb6f2a-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.859673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.859650 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkg8\" (UniqueName: \"kubernetes.io/projected/da100cec-8aca-4cba-b19d-4c873acb6f2a-kube-api-access-mtkg8\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jfdbw\" (UID: \"da100cec-8aca-4cba-b19d-4c873acb6f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:25.963256 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:25.963179 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:26.289741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:26.289655 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw"] Oct 30 13:47:26.293946 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:47:26.293909 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda100cec_8aca_4cba_b19d_4c873acb6f2a.slice/crio-3ebb876b8a203a4cbb693a6d22800830b20bd3ff71b909b4ba8fbed9abbd1502 WatchSource:0}: Error finding container 3ebb876b8a203a4cbb693a6d22800830b20bd3ff71b909b4ba8fbed9abbd1502: Status 404 returned error can't find the container with id 3ebb876b8a203a4cbb693a6d22800830b20bd3ff71b909b4ba8fbed9abbd1502 Oct 30 13:47:26.296146 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:26.296115 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:47:26.296233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:26.296176 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:47:26.296233 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:26.296206 5049 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:47:26.540893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:26.540787 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" event={"ID":"da100cec-8aca-4cba-b19d-4c873acb6f2a","Type":"ContainerStarted","Data":"9a8930528a8ded399f540385949617f0de622952c2d05172ca07062fb112d369"} Oct 30 13:47:26.540893 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:26.540852 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" event={"ID":"da100cec-8aca-4cba-b19d-4c873acb6f2a","Type":"ContainerStarted","Data":"3ebb876b8a203a4cbb693a6d22800830b20bd3ff71b909b4ba8fbed9abbd1502"} Oct 30 13:47:26.586289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:26.586235 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" podStartSLOduration=1.5862153110000001 podStartE2EDuration="1.586215311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:26.583026411 +0000 UTC m=+3353.398220844" watchObservedRunningTime="2026-10-30 13:47:26.586215311 +0000 UTC m=+3353.401409734" Oct 30 13:47:26.682705 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:47:26.682663 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:47:26.966341 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:26.964055 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:27.972346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:27.969462 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jfdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body= Oct 30 13:47:27.972346 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:27.969539 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" podUID="da100cec-8aca-4cba-b19d-4c873acb6f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" Oct 30 13:47:28.963983 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:28.963941 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jfdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body= Oct 30 13:47:28.964204 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:28.964024 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" podUID="da100cec-8aca-4cba-b19d-4c873acb6f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" Oct 30 13:47:29.967852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:29.965692 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jfdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body= Oct 30 13:47:29.967852 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:29.965760 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" podUID="da100cec-8aca-4cba-b19d-4c873acb6f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" Oct 30 13:47:30.637789 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.637749 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz"] Oct 30 13:47:30.642316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.642287 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.643126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.643092 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5vjzz\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.643251 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.643143 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5vjzz\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.643408 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.643275 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24rh7\" (UniqueName: \"kubernetes.io/projected/04c48ac8-b089-4f0c-8ba4-712cd843b70a-kube-api-access-24rh7\") pod \"maas-api-revoke-sub-2e1f5b5f-5vjzz\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.655525 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.655497 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:47:30.665602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.665574 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:47:30.665752 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.665658 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:47:30.666274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.666220 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:47:30.669010 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.668978 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz"] Oct 30 13:47:30.743928 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.743889 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5vjzz\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.744087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.743942 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5vjzz\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.744087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.743993 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24rh7\" (UniqueName: \"kubernetes.io/projected/04c48ac8-b089-4f0c-8ba4-712cd843b70a-kube-api-access-24rh7\") pod \"maas-api-revoke-sub-2e1f5b5f-5vjzz\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.744932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.744906 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5vjzz\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.746514 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.746493 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5vjzz\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.767068 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.767042 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24rh7\" (UniqueName: \"kubernetes.io/projected/04c48ac8-b089-4f0c-8ba4-712cd843b70a-kube-api-access-24rh7\") pod \"maas-api-revoke-sub-2e1f5b5f-5vjzz\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.938758 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.938676 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:30.956801 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.956775 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:30.964159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.964125 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jfdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body= Oct 30 13:47:30.964271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:30.964193 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" podUID="da100cec-8aca-4cba-b19d-4c873acb6f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" Oct 30 13:47:31.277884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:31.277807 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz"] Oct 30 13:47:31.281720 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:47:31.281692 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c48ac8_b089_4f0c_8ba4_712cd843b70a.slice/crio-d9fa6ce88ffbbab492c9ed031344c0267c5173c3c783a198891428a406faf05f WatchSource:0}: Error finding container d9fa6ce88ffbbab492c9ed031344c0267c5173c3c783a198891428a406faf05f: Status 404 returned error can't find the container with id d9fa6ce88ffbbab492c9ed031344c0267c5173c3c783a198891428a406faf05f Oct 30 13:47:31.603111 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:31.603070 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" event={"ID":"04c48ac8-b089-4f0c-8ba4-712cd843b70a","Type":"ContainerStarted","Data":"d9fa6ce88ffbbab492c9ed031344c0267c5173c3c783a198891428a406faf05f"} Oct 30 13:47:31.964556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:31.964473 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jfdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body= Oct 30 13:47:31.964556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:31.964524 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" podUID="da100cec-8aca-4cba-b19d-4c873acb6f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" Oct 30 13:47:32.611036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:32.610950 5049 generic.go:356] "Generic (PLEG): container finished" podID="04c48ac8-b089-4f0c-8ba4-712cd843b70a" containerID="578c35c2646d91fd65334a792ab8b1bb0c58ec2a29fd4a0677fbdc2bd12b2fed" exitCode=0 Oct 30 13:47:32.611036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:32.611005 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" event={"ID":"04c48ac8-b089-4f0c-8ba4-712cd843b70a","Type":"ContainerDied","Data":"578c35c2646d91fd65334a792ab8b1bb0c58ec2a29fd4a0677fbdc2bd12b2fed"} Oct 30 13:47:32.668020 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:32.667984 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:32.964334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:32.964237 5049 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jfdbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" start-of-body= Oct 30 13:47:32.964334 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:32.964296 5049 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" podUID="da100cec-8aca-4cba-b19d-4c873acb6f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.115:15021/healthz/ready\": dial tcp 10.132.1.115:15021: connect: connection refused" Oct 30 13:47:33.756084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:33.756058 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:33.768255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:33.768225 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c48ac8-b089-4f0c-8ba4-712cd843b70a-kube-api-access-24rh7\" (UniqueName: \"kubernetes.io/projected/04c48ac8-b089-4f0c-8ba4-712cd843b70a-kube-api-access-24rh7\") pod \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " Oct 30 13:47:33.768413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:33.768293 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-xks-ca\") pod \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " Oct 30 13:47:33.768498 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:33.768428 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-openshift-ca\") pod \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\" (UID: \"04c48ac8-b089-4f0c-8ba4-712cd843b70a\") " Oct 30 13:47:33.768832 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:33.768798 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-openshift-ca" pod "04c48ac8-b089-4f0c-8ba4-712cd843b70a" (UID: "04c48ac8-b089-4f0c-8ba4-712cd843b70a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:47:33.770829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:33.770804 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-xks-ca" pod "04c48ac8-b089-4f0c-8ba4-712cd843b70a" (UID: "04c48ac8-b089-4f0c-8ba4-712cd843b70a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:47:33.770915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:33.770854 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c48ac8-b089-4f0c-8ba4-712cd843b70a-kube-api-access-24rh7" pod "04c48ac8-b089-4f0c-8ba4-712cd843b70a" (UID: "04c48ac8-b089-4f0c-8ba4-712cd843b70a"). InnerVolumeSpecName "kube-api-access-24rh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:47:33.869673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:33.869592 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:47:33.869673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:33.869622 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24rh7\" (UniqueName: \"kubernetes.io/projected/04c48ac8-b089-4f0c-8ba4-712cd843b70a-kube-api-access-24rh7\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:47:33.869673 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:33.869634 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c48ac8-b089-4f0c-8ba4-712cd843b70a-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:47:34.571550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:34.571517 5049 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:34.571944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:34.571919 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:34.572781 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:34.572757 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jfdbw" Oct 30 13:47:34.626180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:34.626152 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" Oct 30 13:47:34.626180 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:34.626176 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz" event={"ID":"04c48ac8-b089-4f0c-8ba4-712cd843b70a","Type":"ContainerDied","Data":"d9fa6ce88ffbbab492c9ed031344c0267c5173c3c783a198891428a406faf05f"} Oct 30 13:47:34.626430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:34.626222 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9fa6ce88ffbbab492c9ed031344c0267c5173c3c783a198891428a406faf05f" Oct 30 13:47:38.428237 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:38.428205 5049 scope.go:122] "RemoveContainer" containerID="86fbe0dd8554a16f1181c3da13107a78c63ab31740701606ab0c5255a76aeeba" Oct 30 13:47:38.437902 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:38.437877 5049 scope.go:122] "RemoveContainer" containerID="8fa778d7720d7f1b390f39a08c2127c097a9b14a3e15b6aeaf078d070145ea63" Oct 30 13:47:38.446604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:38.446586 5049 scope.go:122] "RemoveContainer" containerID="6dc6944b56815f37423d8daa88acbd105866697adef974403ae65b2b4b82b5c4" Oct 30 13:47:41.757711 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.757670 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz"] Oct 30 13:47:41.758152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.758138 5049 cpu_manager.go:414] "containerMap: removing container" podUID="04c48ac8-b089-4f0c-8ba4-712cd843b70a" containerName="revoke-keys" Oct 30 13:47:41.758195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.758155 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c48ac8-b089-4f0c-8ba4-712cd843b70a" containerName="revoke-keys" Oct 30 13:47:41.758271 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.758262 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c48ac8-b089-4f0c-8ba4-712cd843b70a" containerName="revoke-keys" Oct 30 13:47:41.763439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.763413 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:41.786481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.786441 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:47:41.786481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.786463 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:47:41.786481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.786444 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:47:41.786738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.786569 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:47:41.790073 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.790045 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz"] Oct 30 13:47:41.838776 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.838732 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rbhlz\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:41.838982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.838792 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssgn9\" (UniqueName: \"kubernetes.io/projected/3f226258-9728-494c-8d84-4876af9fd10b-kube-api-access-ssgn9\") pod \"maas-api-revoke-sub-ec843018-rbhlz\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:41.838982 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.838884 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rbhlz\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:41.939740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.939701 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rbhlz\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:41.939942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.939752 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssgn9\" (UniqueName: \"kubernetes.io/projected/3f226258-9728-494c-8d84-4876af9fd10b-kube-api-access-ssgn9\") pod \"maas-api-revoke-sub-ec843018-rbhlz\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:41.939942 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.939784 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rbhlz\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:41.940530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.940504 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rbhlz\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:41.942079 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.942056 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rbhlz\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:41.961083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:41.961052 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssgn9\" (UniqueName: \"kubernetes.io/projected/3f226258-9728-494c-8d84-4876af9fd10b-kube-api-access-ssgn9\") pod \"maas-api-revoke-sub-ec843018-rbhlz\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:42.077534 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:42.077497 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:42.386436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:42.386346 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz"] Oct 30 13:47:42.391740 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:47:42.391702 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f226258_9728_494c_8d84_4876af9fd10b.slice/crio-66da51672844b2efbe691329129ef4639db66dcbef7703d695a4a5ea11b3d7d7 WatchSource:0}: Error finding container 66da51672844b2efbe691329129ef4639db66dcbef7703d695a4a5ea11b3d7d7: Status 404 returned error can't find the container with id 66da51672844b2efbe691329129ef4639db66dcbef7703d695a4a5ea11b3d7d7 Oct 30 13:47:42.677867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:42.677770 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" event={"ID":"3f226258-9728-494c-8d84-4876af9fd10b","Type":"ContainerStarted","Data":"66da51672844b2efbe691329129ef4639db66dcbef7703d695a4a5ea11b3d7d7"} Oct 30 13:47:43.686097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:43.686007 5049 generic.go:356] "Generic (PLEG): container finished" podID="3f226258-9728-494c-8d84-4876af9fd10b" containerID="d70b7669d315dcff7c46bbd4498ea68873eba6f38b1b613e8a7a2d074fc42d9f" exitCode=0 Oct 30 13:47:43.686097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:43.686056 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" event={"ID":"3f226258-9728-494c-8d84-4876af9fd10b","Type":"ContainerDied","Data":"d70b7669d315dcff7c46bbd4498ea68873eba6f38b1b613e8a7a2d074fc42d9f"} Oct 30 13:47:44.876153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:44.876119 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:44.968727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:44.968617 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-openshift-ca\") pod \"3f226258-9728-494c-8d84-4876af9fd10b\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " Oct 30 13:47:44.968727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:44.968676 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-xks-ca\") pod \"3f226258-9728-494c-8d84-4876af9fd10b\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " Oct 30 13:47:44.968727 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:44.968720 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f226258-9728-494c-8d84-4876af9fd10b-kube-api-access-ssgn9\" (UniqueName: \"kubernetes.io/projected/3f226258-9728-494c-8d84-4876af9fd10b-kube-api-access-ssgn9\") pod \"3f226258-9728-494c-8d84-4876af9fd10b\" (UID: \"3f226258-9728-494c-8d84-4876af9fd10b\") " Oct 30 13:47:44.969881 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:44.969854 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-openshift-ca" pod "3f226258-9728-494c-8d84-4876af9fd10b" (UID: "3f226258-9728-494c-8d84-4876af9fd10b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:47:44.982513 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:44.982484 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f226258-9728-494c-8d84-4876af9fd10b-kube-api-access-ssgn9" pod "3f226258-9728-494c-8d84-4876af9fd10b" (UID: "3f226258-9728-494c-8d84-4876af9fd10b"). InnerVolumeSpecName "kube-api-access-ssgn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:47:44.985442 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:44.985412 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-xks-ca" pod "3f226258-9728-494c-8d84-4876af9fd10b" (UID: "3f226258-9728-494c-8d84-4876af9fd10b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:47:45.069863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:45.069812 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:47:45.069863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:45.069863 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f226258-9728-494c-8d84-4876af9fd10b-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:47:45.070161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:45.069879 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssgn9\" (UniqueName: \"kubernetes.io/projected/3f226258-9728-494c-8d84-4876af9fd10b-kube-api-access-ssgn9\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:47:45.707084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:45.706969 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" event={"ID":"3f226258-9728-494c-8d84-4876af9fd10b","Type":"ContainerDied","Data":"66da51672844b2efbe691329129ef4639db66dcbef7703d695a4a5ea11b3d7d7"} Oct 30 13:47:45.707375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:45.707353 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66da51672844b2efbe691329129ef4639db66dcbef7703d695a4a5ea11b3d7d7" Oct 30 13:47:45.707692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:45.707676 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz" Oct 30 13:47:54.164966 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:47:54.164936 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:56.678514 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:47:56.678473 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:48:04.642307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:04.642275 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:04.802638 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:04.802603 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:09.798149 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.798113 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn"] Oct 30 13:48:09.798584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.798554 5049 cpu_manager.go:414] "containerMap: removing container" podUID="3f226258-9728-494c-8d84-4876af9fd10b" containerName="revoke-keys" Oct 30 13:48:09.798584 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.798566 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f226258-9728-494c-8d84-4876af9fd10b" containerName="revoke-keys" Oct 30 13:48:09.798666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.798645 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f226258-9728-494c-8d84-4876af9fd10b" containerName="revoke-keys" Oct 30 13:48:09.802161 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.802142 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:09.815221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.815195 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:48:09.815416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.815248 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:48:09.825223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.825196 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:48:09.825382 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.825230 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:48:09.828158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.828135 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn"] Oct 30 13:48:09.959952 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.959922 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-ldftn\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:09.960141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.959984 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27279\" (UniqueName: \"kubernetes.io/projected/f9140a58-764e-4c41-b031-0f688e95aa26-kube-api-access-27279\") pod \"maas-api-revoke-sub-046e1343-ldftn\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:09.960141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:09.960055 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-ldftn\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:10.061642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:10.061531 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27279\" (UniqueName: \"kubernetes.io/projected/f9140a58-764e-4c41-b031-0f688e95aa26-kube-api-access-27279\") pod \"maas-api-revoke-sub-046e1343-ldftn\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:10.061978 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:10.061944 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-ldftn\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:10.062195 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:10.062176 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-ldftn\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:10.063188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:10.063160 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-ldftn\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:10.066343 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:10.066222 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-ldftn\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:10.083563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:10.083538 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27279\" (UniqueName: \"kubernetes.io/projected/f9140a58-764e-4c41-b031-0f688e95aa26-kube-api-access-27279\") pod \"maas-api-revoke-sub-046e1343-ldftn\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:10.116001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:10.115972 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:10.417140 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:10.417054 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn"] Oct 30 13:48:10.421427 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:48:10.421395 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9140a58_764e_4c41_b031_0f688e95aa26.slice/crio-8a60b2989995f492f9b5242f119b59b59e9f3f0ebd8f768963be9dd4a1979ca5 WatchSource:0}: Error finding container 8a60b2989995f492f9b5242f119b59b59e9f3f0ebd8f768963be9dd4a1979ca5: Status 404 returned error can't find the container with id 8a60b2989995f492f9b5242f119b59b59e9f3f0ebd8f768963be9dd4a1979ca5 Oct 30 13:48:10.965930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:10.965835 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" event={"ID":"f9140a58-764e-4c41-b031-0f688e95aa26","Type":"ContainerStarted","Data":"d796efe1309624abed24d429438229f6f41f8434f35c8164bd0ff12c0abfd625"} Oct 30 13:48:10.965930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:10.965895 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" event={"ID":"f9140a58-764e-4c41-b031-0f688e95aa26","Type":"ContainerStarted","Data":"8a60b2989995f492f9b5242f119b59b59e9f3f0ebd8f768963be9dd4a1979ca5"} Oct 30 13:48:11.004098 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:11.004045 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" podStartSLOduration=2.004032855 podStartE2EDuration="2.004032855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:11.002072195 +0000 UTC m=+3397.817266618" watchObservedRunningTime="2026-10-30 13:48:11.004032855 +0000 UTC m=+3397.819227277" Oct 30 13:48:11.973339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:11.973284 5049 generic.go:356] "Generic (PLEG): container finished" podID="f9140a58-764e-4c41-b031-0f688e95aa26" containerID="d796efe1309624abed24d429438229f6f41f8434f35c8164bd0ff12c0abfd625" exitCode=0 Oct 30 13:48:11.973740 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:11.973373 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" event={"ID":"f9140a58-764e-4c41-b031-0f688e95aa26","Type":"ContainerDied","Data":"d796efe1309624abed24d429438229f6f41f8434f35c8164bd0ff12c0abfd625"} Oct 30 13:48:12.814580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:12.814547 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:13.107131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.107102 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:13.193439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.193407 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-xks-ca\") pod \"f9140a58-764e-4c41-b031-0f688e95aa26\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " Oct 30 13:48:13.193635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.193524 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9140a58-764e-4c41-b031-0f688e95aa26-kube-api-access-27279\" (UniqueName: \"kubernetes.io/projected/f9140a58-764e-4c41-b031-0f688e95aa26-kube-api-access-27279\") pod \"f9140a58-764e-4c41-b031-0f688e95aa26\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " Oct 30 13:48:13.193635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.193628 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-openshift-ca\") pod \"f9140a58-764e-4c41-b031-0f688e95aa26\" (UID: \"f9140a58-764e-4c41-b031-0f688e95aa26\") " Oct 30 13:48:13.194014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.193989 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-openshift-ca" pod "f9140a58-764e-4c41-b031-0f688e95aa26" (UID: "f9140a58-764e-4c41-b031-0f688e95aa26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:48:13.195679 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.195646 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9140a58-764e-4c41-b031-0f688e95aa26-kube-api-access-27279" pod "f9140a58-764e-4c41-b031-0f688e95aa26" (UID: "f9140a58-764e-4c41-b031-0f688e95aa26"). InnerVolumeSpecName "kube-api-access-27279". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:48:13.195797 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.195755 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-xks-ca" pod "f9140a58-764e-4c41-b031-0f688e95aa26" (UID: "f9140a58-764e-4c41-b031-0f688e95aa26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:48:13.295314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.295281 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27279\" (UniqueName: \"kubernetes.io/projected/f9140a58-764e-4c41-b031-0f688e95aa26-kube-api-access-27279\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:48:13.295314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.295310 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:48:13.295314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.295338 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9140a58-764e-4c41-b031-0f688e95aa26-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:48:13.987387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.987359 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" Oct 30 13:48:13.987387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.987373 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn" event={"ID":"f9140a58-764e-4c41-b031-0f688e95aa26","Type":"ContainerDied","Data":"8a60b2989995f492f9b5242f119b59b59e9f3f0ebd8f768963be9dd4a1979ca5"} Oct 30 13:48:13.987631 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:13.987420 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a60b2989995f492f9b5242f119b59b59e9f3f0ebd8f768963be9dd4a1979ca5" Oct 30 13:48:20.113119 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.113081 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb"] Oct 30 13:48:20.113563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.113545 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f9140a58-764e-4c41-b031-0f688e95aa26" containerName="revoke-keys" Oct 30 13:48:20.113563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.113558 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9140a58-764e-4c41-b031-0f688e95aa26" containerName="revoke-keys" Oct 30 13:48:20.113659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.113648 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9140a58-764e-4c41-b031-0f688e95aa26" containerName="revoke-keys" Oct 30 13:48:20.117134 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.117112 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.129887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.129862 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:48:20.140240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.140214 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:48:20.140430 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.140219 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:48:20.140505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.140224 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:48:20.144303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.144274 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb"] Oct 30 13:48:20.167750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.167712 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7jzlb\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.167911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.167845 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7jzlb\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.167964 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.167914 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbrl\" (UniqueName: \"kubernetes.io/projected/d6f36c9d-7049-4473-93aa-2eec11c72caf-kube-api-access-rpbrl\") pod \"maas-api-revoke-sub-d3fba056-7jzlb\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.269197 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.269165 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7jzlb\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.269439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.269239 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7jzlb\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.269439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.269282 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbrl\" (UniqueName: \"kubernetes.io/projected/d6f36c9d-7049-4473-93aa-2eec11c72caf-kube-api-access-rpbrl\") pod \"maas-api-revoke-sub-d3fba056-7jzlb\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.270018 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.269993 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7jzlb\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.271770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.271747 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7jzlb\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.291962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.291933 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbrl\" (UniqueName: \"kubernetes.io/projected/d6f36c9d-7049-4473-93aa-2eec11c72caf-kube-api-access-rpbrl\") pod \"maas-api-revoke-sub-d3fba056-7jzlb\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.430080 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.430003 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:20.737871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:20.737776 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb"] Oct 30 13:48:20.741994 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:48:20.741961 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f36c9d_7049_4473_93aa_2eec11c72caf.slice/crio-413aacdf55b56e6ca3a09b23549aa8b615fb91db74042298d8bd63f23e64df0f WatchSource:0}: Error finding container 413aacdf55b56e6ca3a09b23549aa8b615fb91db74042298d8bd63f23e64df0f: Status 404 returned error can't find the container with id 413aacdf55b56e6ca3a09b23549aa8b615fb91db74042298d8bd63f23e64df0f Oct 30 13:48:21.035480 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:21.035449 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" event={"ID":"d6f36c9d-7049-4473-93aa-2eec11c72caf","Type":"ContainerStarted","Data":"b7accb1dd78db8d1c7ccff0edf976893b0cdf2e1ef7ef9436e07bdc5f9beba22"} Oct 30 13:48:21.035624 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:21.035504 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" event={"ID":"d6f36c9d-7049-4473-93aa-2eec11c72caf","Type":"ContainerStarted","Data":"413aacdf55b56e6ca3a09b23549aa8b615fb91db74042298d8bd63f23e64df0f"} Oct 30 13:48:21.073837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:21.073775 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" podStartSLOduration=1.073759239 podStartE2EDuration="1.073759239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:21.071358209 +0000 UTC m=+3407.886552626" watchObservedRunningTime="2026-10-30 13:48:21.073759239 +0000 UTC m=+3407.888953659" Oct 30 13:48:22.046234 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:22.046190 5049 generic.go:356] "Generic (PLEG): container finished" podID="d6f36c9d-7049-4473-93aa-2eec11c72caf" containerID="b7accb1dd78db8d1c7ccff0edf976893b0cdf2e1ef7ef9436e07bdc5f9beba22" exitCode=0 Oct 30 13:48:22.046761 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:22.046277 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" event={"ID":"d6f36c9d-7049-4473-93aa-2eec11c72caf","Type":"ContainerDied","Data":"b7accb1dd78db8d1c7ccff0edf976893b0cdf2e1ef7ef9436e07bdc5f9beba22"} Oct 30 13:48:23.582549 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:23.582517 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:23.680671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:23.680634 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f36c9d-7049-4473-93aa-2eec11c72caf-kube-api-access-rpbrl\" (UniqueName: \"kubernetes.io/projected/d6f36c9d-7049-4473-93aa-2eec11c72caf-kube-api-access-rpbrl\") pod \"d6f36c9d-7049-4473-93aa-2eec11c72caf\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " Oct 30 13:48:23.680957 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:23.680936 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-xks-ca\") pod \"d6f36c9d-7049-4473-93aa-2eec11c72caf\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " Oct 30 13:48:23.681185 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:23.681167 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-openshift-ca\") pod \"d6f36c9d-7049-4473-93aa-2eec11c72caf\" (UID: \"d6f36c9d-7049-4473-93aa-2eec11c72caf\") " Oct 30 13:48:23.693439 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:23.690996 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-openshift-ca" pod "d6f36c9d-7049-4473-93aa-2eec11c72caf" (UID: "d6f36c9d-7049-4473-93aa-2eec11c72caf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:48:23.702989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:23.702865 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f36c9d-7049-4473-93aa-2eec11c72caf-kube-api-access-rpbrl" pod "d6f36c9d-7049-4473-93aa-2eec11c72caf" (UID: "d6f36c9d-7049-4473-93aa-2eec11c72caf"). InnerVolumeSpecName "kube-api-access-rpbrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:48:23.704878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:23.704849 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-xks-ca" pod "d6f36c9d-7049-4473-93aa-2eec11c72caf" (UID: "d6f36c9d-7049-4473-93aa-2eec11c72caf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:48:23.783175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:23.783013 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbrl\" (UniqueName: \"kubernetes.io/projected/d6f36c9d-7049-4473-93aa-2eec11c72caf-kube-api-access-rpbrl\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:48:23.783175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:23.783056 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:48:23.783175 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:23.783072 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f36c9d-7049-4473-93aa-2eec11c72caf-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:48:24.081806 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:24.081772 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" event={"ID":"d6f36c9d-7049-4473-93aa-2eec11c72caf","Type":"ContainerDied","Data":"413aacdf55b56e6ca3a09b23549aa8b615fb91db74042298d8bd63f23e64df0f"} Oct 30 13:48:24.082004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:24.081808 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb" Oct 30 13:48:24.082004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:24.081818 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="413aacdf55b56e6ca3a09b23549aa8b615fb91db74042298d8bd63f23e64df0f" Oct 30 13:48:26.676293 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:48:26.676256 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:48:31.036239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:31.036198 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:31.193244 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:31.193213 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:38.480338 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:38.480286 5049 scope.go:122] "RemoveContainer" containerID="d40438c6b71243c4fbd7406aede3faa55b4136880c108b7a919d53845a6526a6" Oct 30 13:48:38.492556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:38.492523 5049 scope.go:122] "RemoveContainer" containerID="77a96e261f88b8aa8bfe2da87d8fcfd6623ecb2c22f662665d025a29bf1d0aa5" Oct 30 13:48:38.504967 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:38.504936 5049 scope.go:122] "RemoveContainer" containerID="509aef79f1a6ad1f10d7aa7bc3e25a966cb6c996d69ba0214c8570e8c7bdf8e7" Oct 30 13:48:38.513506 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:38.513481 5049 scope.go:122] "RemoveContainer" containerID="150b49ff26b9f5e214d11c82dab2a6fbf560d6c29a86b0357683b81192d85514" Oct 30 13:48:38.523267 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:38.523245 5049 scope.go:122] "RemoveContainer" containerID="27616f09e1b9fb7a08488b50460d67eb0755b5d2ad0fff83d3650f7bc186c303" Oct 30 13:48:38.531870 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:38.531846 5049 scope.go:122] "RemoveContainer" containerID="5abbf456f2c7464e0e987b560aaf95668b9abe8412583fe7a6dd6c47a01e93ab" Oct 30 13:48:38.540962 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:38.540939 5049 scope.go:122] "RemoveContainer" containerID="05b1b9137b3f77b87aaedd142d339c6fefb964e5f6efd08c933aff769bea8158" Oct 30 13:48:38.551792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:38.551764 5049 scope.go:122] "RemoveContainer" containerID="0910571e1f3a84d0756ef582381d2a89d3b744b432c849960e1a1510a03e9dcb" Oct 30 13:48:41.620184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.620152 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl"] Oct 30 13:48:41.620645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.620630 5049 cpu_manager.go:414] "containerMap: removing container" podUID="d6f36c9d-7049-4473-93aa-2eec11c72caf" containerName="revoke-keys" Oct 30 13:48:41.620701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.620646 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f36c9d-7049-4473-93aa-2eec11c72caf" containerName="revoke-keys" Oct 30 13:48:41.620738 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.620728 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f36c9d-7049-4473-93aa-2eec11c72caf" containerName="revoke-keys" Oct 30 13:48:41.625764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.625742 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:41.638984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.638962 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:48:41.649036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.649009 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:48:41.649177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.649153 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:48:41.649293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.649189 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:48:41.651921 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.651896 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl"] Oct 30 13:48:41.818404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.818368 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6dbvl\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:41.818404 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.818410 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b856w\" (UniqueName: \"kubernetes.io/projected/8b54735c-2985-438d-b5d7-c169a21a6c56-kube-api-access-b856w\") pod \"maas-api-revoke-sub-ae594979-6dbvl\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:41.818695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.818587 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6dbvl\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:41.919359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.919226 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6dbvl\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:41.919359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.919282 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b856w\" (UniqueName: \"kubernetes.io/projected/8b54735c-2985-438d-b5d7-c169a21a6c56-kube-api-access-b856w\") pod \"maas-api-revoke-sub-ae594979-6dbvl\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:41.919635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.919394 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6dbvl\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:41.920035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.920008 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6dbvl\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:41.921692 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.921674 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6dbvl\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:41.941652 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:41.941623 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b856w\" (UniqueName: \"kubernetes.io/projected/8b54735c-2985-438d-b5d7-c169a21a6c56-kube-api-access-b856w\") pod \"maas-api-revoke-sub-ae594979-6dbvl\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:42.239495 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:42.239413 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:42.542473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:42.542394 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl"] Oct 30 13:48:42.546223 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:48:42.546198 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b54735c_2985_438d_b5d7_c169a21a6c56.slice/crio-d35c8a16bed6f81f1643709fcd6ab25abb1e6eb60731e228d52aa7109466e5ca WatchSource:0}: Error finding container d35c8a16bed6f81f1643709fcd6ab25abb1e6eb60731e228d52aa7109466e5ca: Status 404 returned error can't find the container with id d35c8a16bed6f81f1643709fcd6ab25abb1e6eb60731e228d52aa7109466e5ca Oct 30 13:48:43.241869 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:43.241828 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" event={"ID":"8b54735c-2985-438d-b5d7-c169a21a6c56","Type":"ContainerStarted","Data":"2cadb7e9c4446f370e1fc5c9fa557eede556b1136015301a6ab1c805ccb6a556"} Oct 30 13:48:43.242275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:43.241893 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" event={"ID":"8b54735c-2985-438d-b5d7-c169a21a6c56","Type":"ContainerStarted","Data":"d35c8a16bed6f81f1643709fcd6ab25abb1e6eb60731e228d52aa7109466e5ca"} Oct 30 13:48:43.282090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:43.282031 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" podStartSLOduration=2.282012121 podStartE2EDuration="2.282012121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:43.279035244 +0000 UTC m=+3430.094229664" watchObservedRunningTime="2026-10-30 13:48:43.282012121 +0000 UTC m=+3430.097206545" Oct 30 13:48:44.249598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:44.249559 5049 generic.go:356] "Generic (PLEG): container finished" podID="8b54735c-2985-438d-b5d7-c169a21a6c56" containerID="2cadb7e9c4446f370e1fc5c9fa557eede556b1136015301a6ab1c805ccb6a556" exitCode=0 Oct 30 13:48:44.250096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:44.249639 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" event={"ID":"8b54735c-2985-438d-b5d7-c169a21a6c56","Type":"ContainerDied","Data":"2cadb7e9c4446f370e1fc5c9fa557eede556b1136015301a6ab1c805ccb6a556"} Oct 30 13:48:45.392807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:45.392776 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:45.452070 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:45.452040 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-openshift-ca\") pod \"8b54735c-2985-438d-b5d7-c169a21a6c56\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " Oct 30 13:48:45.452268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:45.452095 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b54735c-2985-438d-b5d7-c169a21a6c56-kube-api-access-b856w\" (UniqueName: \"kubernetes.io/projected/8b54735c-2985-438d-b5d7-c169a21a6c56-kube-api-access-b856w\") pod \"8b54735c-2985-438d-b5d7-c169a21a6c56\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " Oct 30 13:48:45.452268 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:45.452121 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-xks-ca\") pod \"8b54735c-2985-438d-b5d7-c169a21a6c56\" (UID: \"8b54735c-2985-438d-b5d7-c169a21a6c56\") " Oct 30 13:48:45.452515 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:45.452491 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-openshift-ca" pod "8b54735c-2985-438d-b5d7-c169a21a6c56" (UID: "8b54735c-2985-438d-b5d7-c169a21a6c56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:48:45.454289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:45.454259 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-xks-ca" pod "8b54735c-2985-438d-b5d7-c169a21a6c56" (UID: "8b54735c-2985-438d-b5d7-c169a21a6c56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:48:45.454289 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:45.454268 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b54735c-2985-438d-b5d7-c169a21a6c56-kube-api-access-b856w" pod "8b54735c-2985-438d-b5d7-c169a21a6c56" (UID: "8b54735c-2985-438d-b5d7-c169a21a6c56"). InnerVolumeSpecName "kube-api-access-b856w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:48:45.553288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:45.553256 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b856w\" (UniqueName: \"kubernetes.io/projected/8b54735c-2985-438d-b5d7-c169a21a6c56-kube-api-access-b856w\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:48:45.553288 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:45.553287 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:48:45.553518 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:45.553296 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b54735c-2985-438d-b5d7-c169a21a6c56-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:48:46.264097 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:46.264067 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" Oct 30 13:48:46.264314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:46.264064 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl" event={"ID":"8b54735c-2985-438d-b5d7-c169a21a6c56","Type":"ContainerDied","Data":"d35c8a16bed6f81f1643709fcd6ab25abb1e6eb60731e228d52aa7109466e5ca"} Oct 30 13:48:46.264314 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:48:46.264196 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35c8a16bed6f81f1643709fcd6ab25abb1e6eb60731e228d52aa7109466e5ca" Oct 30 13:48:56.741127 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:48:56.741070 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:49:03.270862 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.270824 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk"] Oct 30 13:49:03.271349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.271275 5049 cpu_manager.go:414] "containerMap: removing container" podUID="8b54735c-2985-438d-b5d7-c169a21a6c56" containerName="revoke-keys" Oct 30 13:49:03.271349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.271287 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b54735c-2985-438d-b5d7-c169a21a6c56" containerName="revoke-keys" Oct 30 13:49:03.271452 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.271415 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b54735c-2985-438d-b5d7-c169a21a6c56" containerName="revoke-keys" Oct 30 13:49:03.275002 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.274981 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.288279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.288255 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:49:03.298829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.298801 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:49:03.299014 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.298803 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:49:03.299139 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.298846 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:49:03.301841 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.301816 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk"] Oct 30 13:49:03.354049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.354018 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h5ssk\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.354227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.354073 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h5ssk\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.354227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.354138 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjhhs\" (UniqueName: \"kubernetes.io/projected/f929ecc6-4b59-4b20-96c6-8a69365eb789-kube-api-access-wjhhs\") pod \"maas-api-revoke-sub-a4aef7d7-h5ssk\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.454899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.454840 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h5ssk\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.454899 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.454915 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h5ssk\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.455135 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.454946 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjhhs\" (UniqueName: \"kubernetes.io/projected/f929ecc6-4b59-4b20-96c6-8a69365eb789-kube-api-access-wjhhs\") pod \"maas-api-revoke-sub-a4aef7d7-h5ssk\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.455678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.455653 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h5ssk\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.457312 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.457286 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h5ssk\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.477476 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.477456 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjhhs\" (UniqueName: \"kubernetes.io/projected/f929ecc6-4b59-4b20-96c6-8a69365eb789-kube-api-access-wjhhs\") pod \"maas-api-revoke-sub-a4aef7d7-h5ssk\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.588106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.588073 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:03.892414 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:03.892313 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk"] Oct 30 13:49:03.895775 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:49:03.895745 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf929ecc6_4b59_4b20_96c6_8a69365eb789.slice/crio-d07effd1908b3a7bec0cb14e4c0cf2a3945e16512ce21fef6fe9425c97975cca WatchSource:0}: Error finding container d07effd1908b3a7bec0cb14e4c0cf2a3945e16512ce21fef6fe9425c97975cca: Status 404 returned error can't find the container with id d07effd1908b3a7bec0cb14e4c0cf2a3945e16512ce21fef6fe9425c97975cca Oct 30 13:49:04.427814 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:04.427776 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" event={"ID":"f929ecc6-4b59-4b20-96c6-8a69365eb789","Type":"ContainerStarted","Data":"2628813fba6c842b60c97751fb11cc85bc7297c5656210ee2039d1c67cfa2459"} Oct 30 13:49:04.428219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:04.427830 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" event={"ID":"f929ecc6-4b59-4b20-96c6-8a69365eb789","Type":"ContainerStarted","Data":"d07effd1908b3a7bec0cb14e4c0cf2a3945e16512ce21fef6fe9425c97975cca"} Oct 30 13:49:04.466475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:04.466394 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" podStartSLOduration=1.466381568 podStartE2EDuration="1.466381568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:04.465850893 +0000 UTC m=+3451.281045320" watchObservedRunningTime="2026-10-30 13:49:04.466381568 +0000 UTC m=+3451.281575990" Oct 30 13:49:05.435879 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:05.435843 5049 generic.go:356] "Generic (PLEG): container finished" podID="f929ecc6-4b59-4b20-96c6-8a69365eb789" containerID="2628813fba6c842b60c97751fb11cc85bc7297c5656210ee2039d1c67cfa2459" exitCode=0 Oct 30 13:49:05.436275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:05.435918 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" event={"ID":"f929ecc6-4b59-4b20-96c6-8a69365eb789","Type":"ContainerDied","Data":"2628813fba6c842b60c97751fb11cc85bc7297c5656210ee2039d1c67cfa2459"} Oct 30 13:49:06.583275 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:06.583249 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:06.687029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:06.686996 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-openshift-ca\") pod \"f929ecc6-4b59-4b20-96c6-8a69365eb789\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " Oct 30 13:49:06.687220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:06.687100 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f929ecc6-4b59-4b20-96c6-8a69365eb789-kube-api-access-wjhhs\" (UniqueName: \"kubernetes.io/projected/f929ecc6-4b59-4b20-96c6-8a69365eb789-kube-api-access-wjhhs\") pod \"f929ecc6-4b59-4b20-96c6-8a69365eb789\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " Oct 30 13:49:06.687220 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:06.687124 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-xks-ca\") pod \"f929ecc6-4b59-4b20-96c6-8a69365eb789\" (UID: \"f929ecc6-4b59-4b20-96c6-8a69365eb789\") " Oct 30 13:49:06.687473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:06.687445 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-openshift-ca" pod "f929ecc6-4b59-4b20-96c6-8a69365eb789" (UID: "f929ecc6-4b59-4b20-96c6-8a69365eb789"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:06.689166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:06.689141 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-xks-ca" pod "f929ecc6-4b59-4b20-96c6-8a69365eb789" (UID: "f929ecc6-4b59-4b20-96c6-8a69365eb789"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:06.689291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:06.689271 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f929ecc6-4b59-4b20-96c6-8a69365eb789-kube-api-access-wjhhs" pod "f929ecc6-4b59-4b20-96c6-8a69365eb789" (UID: "f929ecc6-4b59-4b20-96c6-8a69365eb789"). InnerVolumeSpecName "kube-api-access-wjhhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:06.788447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:06.788412 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:06.788447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:06.788442 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjhhs\" (UniqueName: \"kubernetes.io/projected/f929ecc6-4b59-4b20-96c6-8a69365eb789-kube-api-access-wjhhs\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:06.788447 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:06.788452 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f929ecc6-4b59-4b20-96c6-8a69365eb789-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:07.449867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:07.449841 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" Oct 30 13:49:07.450021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:07.449883 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk" event={"ID":"f929ecc6-4b59-4b20-96c6-8a69365eb789","Type":"ContainerDied","Data":"d07effd1908b3a7bec0cb14e4c0cf2a3945e16512ce21fef6fe9425c97975cca"} Oct 30 13:49:07.450021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:07.449931 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d07effd1908b3a7bec0cb14e4c0cf2a3945e16512ce21fef6fe9425c97975cca" Oct 30 13:49:08.109169 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:08.109132 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:23.609003 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.608917 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj"] Oct 30 13:49:23.609578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.609401 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f929ecc6-4b59-4b20-96c6-8a69365eb789" containerName="revoke-keys" Oct 30 13:49:23.609578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.609414 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f929ecc6-4b59-4b20-96c6-8a69365eb789" containerName="revoke-keys" Oct 30 13:49:23.609578 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.609524 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f929ecc6-4b59-4b20-96c6-8a69365eb789" containerName="revoke-keys" Oct 30 13:49:23.612907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.612887 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:23.625944 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.625918 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:49:23.636129 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.636102 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:49:23.636281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.636150 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:49:23.636281 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.636191 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:49:23.639726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.639658 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj"] Oct 30 13:49:23.694755 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.694722 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9f8gj\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:23.694930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.694768 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9f8gj\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:23.694930 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.694874 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gth8x\" (UniqueName: \"kubernetes.io/projected/ec1a4838-d99a-41b6-a45c-900ecefe5583-kube-api-access-gth8x\") pod \"maas-api-revoke-sub-79805ca9-9f8gj\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:23.796264 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.796227 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gth8x\" (UniqueName: \"kubernetes.io/projected/ec1a4838-d99a-41b6-a45c-900ecefe5583-kube-api-access-gth8x\") pod \"maas-api-revoke-sub-79805ca9-9f8gj\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:23.796464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.796348 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9f8gj\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:23.796464 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.796375 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9f8gj\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:23.797120 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.797092 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9f8gj\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:23.798707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.798683 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9f8gj\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:23.817807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.817778 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gth8x\" (UniqueName: \"kubernetes.io/projected/ec1a4838-d99a-41b6-a45c-900ecefe5583-kube-api-access-gth8x\") pod \"maas-api-revoke-sub-79805ca9-9f8gj\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:23.926535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:23.926447 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:24.229210 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:24.229130 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj"] Oct 30 13:49:24.233457 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:49:24.233429 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1a4838_d99a_41b6_a45c_900ecefe5583.slice/crio-06676a40110b616e62295859fcdd8ae6d21b8ac1cb23ebbd9e9b25e93ce6ad38 WatchSource:0}: Error finding container 06676a40110b616e62295859fcdd8ae6d21b8ac1cb23ebbd9e9b25e93ce6ad38: Status 404 returned error can't find the container with id 06676a40110b616e62295859fcdd8ae6d21b8ac1cb23ebbd9e9b25e93ce6ad38 Oct 30 13:49:24.605016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:24.604982 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" event={"ID":"ec1a4838-d99a-41b6-a45c-900ecefe5583","Type":"ContainerStarted","Data":"01cb1049b2df3cad2cbe8bf9a7659b75f1c9e1fa00213ba3b0b6f571aab9af5b"} Oct 30 13:49:24.605203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:24.605033 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" event={"ID":"ec1a4838-d99a-41b6-a45c-900ecefe5583","Type":"ContainerStarted","Data":"06676a40110b616e62295859fcdd8ae6d21b8ac1cb23ebbd9e9b25e93ce6ad38"} Oct 30 13:49:24.642291 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:24.642236 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" podStartSLOduration=1.6422224029999999 podStartE2EDuration="1.642222403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:24.641702593 +0000 UTC m=+3471.456897016" watchObservedRunningTime="2026-10-30 13:49:24.642222403 +0000 UTC m=+3471.457416825" Oct 30 13:49:25.613276 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:25.613239 5049 generic.go:356] "Generic (PLEG): container finished" podID="ec1a4838-d99a-41b6-a45c-900ecefe5583" containerID="01cb1049b2df3cad2cbe8bf9a7659b75f1c9e1fa00213ba3b0b6f571aab9af5b" exitCode=0 Oct 30 13:49:25.613500 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:25.613347 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" event={"ID":"ec1a4838-d99a-41b6-a45c-900ecefe5583","Type":"ContainerDied","Data":"01cb1049b2df3cad2cbe8bf9a7659b75f1c9e1fa00213ba3b0b6f571aab9af5b"} Oct 30 13:49:26.682610 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:49:26.682553 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:49:26.756890 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:26.756865 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:26.929284 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:26.929203 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-openshift-ca\") pod \"ec1a4838-d99a-41b6-a45c-900ecefe5583\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " Oct 30 13:49:26.929453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:26.929352 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-xks-ca\") pod \"ec1a4838-d99a-41b6-a45c-900ecefe5583\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " Oct 30 13:49:26.929453 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:26.929417 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1a4838-d99a-41b6-a45c-900ecefe5583-kube-api-access-gth8x\" (UniqueName: \"kubernetes.io/projected/ec1a4838-d99a-41b6-a45c-900ecefe5583-kube-api-access-gth8x\") pod \"ec1a4838-d99a-41b6-a45c-900ecefe5583\" (UID: \"ec1a4838-d99a-41b6-a45c-900ecefe5583\") " Oct 30 13:49:26.929580 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:26.929556 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-openshift-ca" pod "ec1a4838-d99a-41b6-a45c-900ecefe5583" (UID: "ec1a4838-d99a-41b6-a45c-900ecefe5583"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:26.929701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:26.929682 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:26.931209 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:26.931184 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-xks-ca" pod "ec1a4838-d99a-41b6-a45c-900ecefe5583" (UID: "ec1a4838-d99a-41b6-a45c-900ecefe5583"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:26.931349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:26.931331 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1a4838-d99a-41b6-a45c-900ecefe5583-kube-api-access-gth8x" pod "ec1a4838-d99a-41b6-a45c-900ecefe5583" (UID: "ec1a4838-d99a-41b6-a45c-900ecefe5583"). InnerVolumeSpecName "kube-api-access-gth8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:27.031247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:27.031211 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gth8x\" (UniqueName: \"kubernetes.io/projected/ec1a4838-d99a-41b6-a45c-900ecefe5583-kube-api-access-gth8x\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:27.031247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:27.031246 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1a4838-d99a-41b6-a45c-900ecefe5583-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:27.627188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:27.627111 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" Oct 30 13:49:27.627188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:27.627138 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj" event={"ID":"ec1a4838-d99a-41b6-a45c-900ecefe5583","Type":"ContainerDied","Data":"06676a40110b616e62295859fcdd8ae6d21b8ac1cb23ebbd9e9b25e93ce6ad38"} Oct 30 13:49:27.627188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:27.627185 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06676a40110b616e62295859fcdd8ae6d21b8ac1cb23ebbd9e9b25e93ce6ad38" Oct 30 13:49:31.149940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:31.149910 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:31.351932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:31.351897 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:36.029568 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:36.029464 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd"] Oct 30 13:49:36.040861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:36.040821 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c2c6ae4-4ktfd"] Oct 30 13:49:36.048669 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:36.048620 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs"] Oct 30 13:49:36.048945 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:36.048919 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0f26a78-27pzs"] Oct 30 13:49:37.370344 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:37.370175 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41593460-aec9-43dd-a4ad-27be23c7b607" path="/var/lib/kubelet/pods/41593460-aec9-43dd-a4ad-27be23c7b607/volumes" Oct 30 13:49:37.370861 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:37.370811 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1" path="/var/lib/kubelet/pods/65ce5b64-cf90-4d4e-84d3-f3d59a1eeef1/volumes" Oct 30 13:49:38.629094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:38.626108 5049 scope.go:122] "RemoveContainer" containerID="48780cf2f95369208e760ede4454241451dc7bbb0362cc92caaea0ae0c027f97" Oct 30 13:49:38.673545 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:38.672930 5049 scope.go:122] "RemoveContainer" containerID="f43a5362c4bbe1842473c3a83bd3d3bdadbc3cd996124045b03afb4a9c4834bd" Oct 30 13:49:38.699196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:38.699158 5049 scope.go:122] "RemoveContainer" containerID="f4dffd41d091dfa01e3846e63eb850b90104396fb574e7564d8d1d0a29b38231" Oct 30 13:49:38.745358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:38.744545 5049 scope.go:122] "RemoveContainer" containerID="ec6b62b640e756be3e0f579ad22c628a2bf312749baa8c2f4c72e125426fa84f" Oct 30 13:49:38.773493 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:38.773461 5049 scope.go:122] "RemoveContainer" containerID="31fb9c331999d7605ea98bab435eaee6d7a4d4b7097234de1fa404f01f2ea1a5" Oct 30 13:49:41.486212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.486171 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk"] Oct 30 13:49:41.486658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.486628 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ec1a4838-d99a-41b6-a45c-900ecefe5583" containerName="revoke-keys" Oct 30 13:49:41.486658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.486640 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1a4838-d99a-41b6-a45c-900ecefe5583" containerName="revoke-keys" Oct 30 13:49:41.486746 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.486723 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1a4838-d99a-41b6-a45c-900ecefe5583" containerName="revoke-keys" Oct 30 13:49:41.491682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.491662 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:41.505046 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.505026 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:49:41.514998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.514959 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:49:41.515178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.515049 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:49:41.515178 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.515078 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:49:41.515366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.515341 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q5xrk\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:41.515473 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.515428 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q5xrk\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:41.515538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.515474 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7hn\" (UniqueName: \"kubernetes.io/projected/023c09b8-1dd6-412e-a2d9-2330236037bd-kube-api-access-gv7hn\") pod \"maas-api-revoke-sub-0b7e8edb-q5xrk\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:41.517998 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.517977 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk"] Oct 30 13:49:41.616590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.616557 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q5xrk\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:41.616771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.616612 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7hn\" (UniqueName: \"kubernetes.io/projected/023c09b8-1dd6-412e-a2d9-2330236037bd-kube-api-access-gv7hn\") pod \"maas-api-revoke-sub-0b7e8edb-q5xrk\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:41.616771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.616664 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q5xrk\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:41.617311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.617288 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q5xrk\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:41.619052 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.619025 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q5xrk\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:41.637866 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.637844 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7hn\" (UniqueName: \"kubernetes.io/projected/023c09b8-1dd6-412e-a2d9-2330236037bd-kube-api-access-gv7hn\") pod \"maas-api-revoke-sub-0b7e8edb-q5xrk\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:41.804274 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:41.804242 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:42.108867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:42.108789 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk"] Oct 30 13:49:42.112742 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:49:42.112710 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023c09b8_1dd6_412e_a2d9_2330236037bd.slice/crio-60ae86d266b3ed6e833a1e5734bc8b91bcbd16140f0a9482a909bbb3f8443122 WatchSource:0}: Error finding container 60ae86d266b3ed6e833a1e5734bc8b91bcbd16140f0a9482a909bbb3f8443122: Status 404 returned error can't find the container with id 60ae86d266b3ed6e833a1e5734bc8b91bcbd16140f0a9482a909bbb3f8443122 Oct 30 13:49:42.728642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:42.728553 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" event={"ID":"023c09b8-1dd6-412e-a2d9-2330236037bd","Type":"ContainerStarted","Data":"149d0b420866976350aca9a21f4d961f7154ad387487d7715c503f455bf2df18"} Oct 30 13:49:42.728642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:42.728634 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" event={"ID":"023c09b8-1dd6-412e-a2d9-2330236037bd","Type":"ContainerStarted","Data":"60ae86d266b3ed6e833a1e5734bc8b91bcbd16140f0a9482a909bbb3f8443122"} Oct 30 13:49:42.766655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:42.766606 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" podStartSLOduration=1.766593815 podStartE2EDuration="1.766593815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:42.765438852 +0000 UTC m=+3489.580633275" watchObservedRunningTime="2026-10-30 13:49:42.766593815 +0000 UTC m=+3489.581788239" Oct 30 13:49:43.735986 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:43.735944 5049 generic.go:356] "Generic (PLEG): container finished" podID="023c09b8-1dd6-412e-a2d9-2330236037bd" containerID="149d0b420866976350aca9a21f4d961f7154ad387487d7715c503f455bf2df18" exitCode=0 Oct 30 13:49:43.736394 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:43.736029 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" event={"ID":"023c09b8-1dd6-412e-a2d9-2330236037bd","Type":"ContainerDied","Data":"149d0b420866976350aca9a21f4d961f7154ad387487d7715c503f455bf2df18"} Oct 30 13:49:44.879703 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:44.879673 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:44.950265 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:44.950227 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023c09b8-1dd6-412e-a2d9-2330236037bd-kube-api-access-gv7hn\" (UniqueName: \"kubernetes.io/projected/023c09b8-1dd6-412e-a2d9-2330236037bd-kube-api-access-gv7hn\") pod \"023c09b8-1dd6-412e-a2d9-2330236037bd\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " Oct 30 13:49:44.950436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:44.950339 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-openshift-ca\") pod \"023c09b8-1dd6-412e-a2d9-2330236037bd\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " Oct 30 13:49:44.950436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:44.950420 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-xks-ca\") pod \"023c09b8-1dd6-412e-a2d9-2330236037bd\" (UID: \"023c09b8-1dd6-412e-a2d9-2330236037bd\") " Oct 30 13:49:44.950649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:44.950627 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-openshift-ca" pod "023c09b8-1dd6-412e-a2d9-2330236037bd" (UID: "023c09b8-1dd6-412e-a2d9-2330236037bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:44.950791 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:44.950776 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:44.952490 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:44.952463 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-xks-ca" pod "023c09b8-1dd6-412e-a2d9-2330236037bd" (UID: "023c09b8-1dd6-412e-a2d9-2330236037bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:44.952606 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:44.952518 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023c09b8-1dd6-412e-a2d9-2330236037bd-kube-api-access-gv7hn" pod "023c09b8-1dd6-412e-a2d9-2330236037bd" (UID: "023c09b8-1dd6-412e-a2d9-2330236037bd"). InnerVolumeSpecName "kube-api-access-gv7hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:45.051315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:45.051285 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023c09b8-1dd6-412e-a2d9-2330236037bd-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:45.051315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:45.051315 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv7hn\" (UniqueName: \"kubernetes.io/projected/023c09b8-1dd6-412e-a2d9-2330236037bd-kube-api-access-gv7hn\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:45.749021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:45.748943 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" Oct 30 13:49:45.749021 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:45.748981 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk" event={"ID":"023c09b8-1dd6-412e-a2d9-2330236037bd","Type":"ContainerDied","Data":"60ae86d266b3ed6e833a1e5734bc8b91bcbd16140f0a9482a909bbb3f8443122"} Oct 30 13:49:45.749212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:45.749029 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60ae86d266b3ed6e833a1e5734bc8b91bcbd16140f0a9482a909bbb3f8443122" Oct 30 13:49:52.280557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.280514 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz"] Oct 30 13:49:52.281007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.280992 5049 cpu_manager.go:414] "containerMap: removing container" podUID="023c09b8-1dd6-412e-a2d9-2330236037bd" containerName="revoke-keys" Oct 30 13:49:52.281049 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.281008 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023c09b8-1dd6-412e-a2d9-2330236037bd" containerName="revoke-keys" Oct 30 13:49:52.281094 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.281079 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="023c09b8-1dd6-412e-a2d9-2330236037bd" containerName="revoke-keys" Oct 30 13:49:52.284552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.284531 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.298794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.298767 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:49:52.308999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.308968 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:49:52.309140 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.309056 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:49:52.309207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.309140 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:49:52.312004 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.311980 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz"] Oct 30 13:49:52.421613 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.421574 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-df4xz\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.421871 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.421845 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/9b998117-6bfa-410a-8e42-293cd4024c7a-kube-api-access-s7jmc\") pod \"maas-api-revoke-sub-3d4156f1-df4xz\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.421989 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.421972 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-df4xz\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.522973 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.522920 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-df4xz\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.523186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.523033 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/9b998117-6bfa-410a-8e42-293cd4024c7a-kube-api-access-s7jmc\") pod \"maas-api-revoke-sub-3d4156f1-df4xz\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.523186 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.523078 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-df4xz\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.523666 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.523642 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-df4xz\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.525530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.525506 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-df4xz\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.545741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.545707 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/9b998117-6bfa-410a-8e42-293cd4024c7a-kube-api-access-s7jmc\") pod \"maas-api-revoke-sub-3d4156f1-df4xz\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.597887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.597851 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:52.906657 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:52.906571 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz"] Oct 30 13:49:52.911010 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:49:52.910979 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b998117_6bfa_410a_8e42_293cd4024c7a.slice/crio-69598fcb99a06b3305caf0b4053b3579c4122edcaf7b0757477032c29e06335b WatchSource:0}: Error finding container 69598fcb99a06b3305caf0b4053b3579c4122edcaf7b0757477032c29e06335b: Status 404 returned error can't find the container with id 69598fcb99a06b3305caf0b4053b3579c4122edcaf7b0757477032c29e06335b Oct 30 13:49:53.798914 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:53.798878 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" event={"ID":"9b998117-6bfa-410a-8e42-293cd4024c7a","Type":"ContainerStarted","Data":"29a3e4f5c74b7ba7ada2fb880936f90442947d900f21e60e932f469c5ecdb1a2"} Oct 30 13:49:53.799373 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:53.798936 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" event={"ID":"9b998117-6bfa-410a-8e42-293cd4024c7a","Type":"ContainerStarted","Data":"69598fcb99a06b3305caf0b4053b3579c4122edcaf7b0757477032c29e06335b"} Oct 30 13:49:53.836508 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:53.836459 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" podStartSLOduration=1.83644557 podStartE2EDuration="1.83644557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:53.835429748 +0000 UTC m=+3500.650624181" watchObservedRunningTime="2026-10-30 13:49:53.83644557 +0000 UTC m=+3500.651639990" Oct 30 13:49:54.806626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:54.806585 5049 generic.go:356] "Generic (PLEG): container finished" podID="9b998117-6bfa-410a-8e42-293cd4024c7a" containerID="29a3e4f5c74b7ba7ada2fb880936f90442947d900f21e60e932f469c5ecdb1a2" exitCode=0 Oct 30 13:49:54.807172 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:54.806667 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" event={"ID":"9b998117-6bfa-410a-8e42-293cd4024c7a","Type":"ContainerDied","Data":"29a3e4f5c74b7ba7ada2fb880936f90442947d900f21e60e932f469c5ecdb1a2"} Oct 30 13:49:55.958642 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:55.958616 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:56.058448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.058407 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-xks-ca\") pod \"9b998117-6bfa-410a-8e42-293cd4024c7a\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " Oct 30 13:49:56.058649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.058490 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b998117-6bfa-410a-8e42-293cd4024c7a-kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/9b998117-6bfa-410a-8e42-293cd4024c7a-kube-api-access-s7jmc\") pod \"9b998117-6bfa-410a-8e42-293cd4024c7a\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " Oct 30 13:49:56.058649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.058558 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-openshift-ca\") pod \"9b998117-6bfa-410a-8e42-293cd4024c7a\" (UID: \"9b998117-6bfa-410a-8e42-293cd4024c7a\") " Oct 30 13:49:56.058938 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.058908 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-openshift-ca" pod "9b998117-6bfa-410a-8e42-293cd4024c7a" (UID: "9b998117-6bfa-410a-8e42-293cd4024c7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:56.060635 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.060614 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b998117-6bfa-410a-8e42-293cd4024c7a-kube-api-access-s7jmc" pod "9b998117-6bfa-410a-8e42-293cd4024c7a" (UID: "9b998117-6bfa-410a-8e42-293cd4024c7a"). InnerVolumeSpecName "kube-api-access-s7jmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:56.060743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.060652 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-xks-ca" pod "9b998117-6bfa-410a-8e42-293cd4024c7a" (UID: "9b998117-6bfa-410a-8e42-293cd4024c7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:56.159651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.159618 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:56.159651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.159646 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/9b998117-6bfa-410a-8e42-293cd4024c7a-kube-api-access-s7jmc\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:56.159651 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.159656 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b998117-6bfa-410a-8e42-293cd4024c7a-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:49:56.677307 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:49:56.677232 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:49:56.819715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.819679 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" Oct 30 13:49:56.819715 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.819710 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz" event={"ID":"9b998117-6bfa-410a-8e42-293cd4024c7a","Type":"ContainerDied","Data":"69598fcb99a06b3305caf0b4053b3579c4122edcaf7b0757477032c29e06335b"} Oct 30 13:49:56.819934 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:49:56.819759 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69598fcb99a06b3305caf0b4053b3579c4122edcaf7b0757477032c29e06335b" Oct 30 13:50:03.027980 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:03.027941 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs"] Oct 30 13:50:03.038188 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:03.038154 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e168c9e7-pcwgs"] Oct 30 13:50:03.346615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:03.346581 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937fd81f-4191-49ed-9a59-216ea01544c5" path="/var/lib/kubelet/pods/937fd81f-4191-49ed-9a59-216ea01544c5/volumes" Oct 30 13:50:07.043067 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:07.043032 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx"] Oct 30 13:50:07.052258 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:07.052230 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac3bc12-jlntx"] Oct 30 13:50:07.054487 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:07.054455 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz"] Oct 30 13:50:07.056370 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:07.056346 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ee1507-fjszz"] Oct 30 13:50:07.334419 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:07.334381 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46821723-84da-4e91-b3d1-e4de9f047b93" path="/var/lib/kubelet/pods/46821723-84da-4e91-b3d1-e4de9f047b93/volumes" Oct 30 13:50:07.334848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:07.334832 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7574c065-ecfa-4afc-b346-2519aa9be554" path="/var/lib/kubelet/pods/7574c065-ecfa-4afc-b346-2519aa9be554/volumes" Oct 30 13:50:09.033218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:09.033187 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k"] Oct 30 13:50:09.037911 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:09.037883 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3feedf6e-fn79k"] Oct 30 13:50:09.335221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:09.335186 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffafa367-dab6-48bb-939f-d84a49d43194" path="/var/lib/kubelet/pods/ffafa367-dab6-48bb-939f-d84a49d43194/volumes" Oct 30 13:50:13.667375 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:13.667341 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz"] Oct 30 13:50:13.677166 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:13.677135 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-df4xz"] Oct 30 13:50:15.334720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:15.334689 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b998117-6bfa-410a-8e42-293cd4024c7a" path="/var/lib/kubelet/pods/9b998117-6bfa-410a-8e42-293cd4024c7a/volumes" Oct 30 13:50:26.677246 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:50:26.677205 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:50:29.042786 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:29.042752 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w"] Oct 30 13:50:29.054215 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:29.054184 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d521cb-55x4w"] Oct 30 13:50:29.334167 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:29.334136 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1e8fae-1c7e-4cca-8ba5-53fdb6740834" path="/var/lib/kubelet/pods/da1e8fae-1c7e-4cca-8ba5-53fdb6740834/volumes" Oct 30 13:50:31.282830 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:31.282791 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:34.186218 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:34.186185 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:38.870481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:38.870446 5049 scope.go:122] "RemoveContainer" containerID="bf67a35dde3b4a64de714b746b018b7ba39a6750d47e961f1a288a4a947b3e0e" Oct 30 13:50:38.880821 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:38.880795 5049 scope.go:122] "RemoveContainer" containerID="2ca4088d7b8c5f4f45eae96fadf32eb3f0c80e2323ab5fd9152e0134a3498595" Oct 30 13:50:38.889915 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:38.889895 5049 scope.go:122] "RemoveContainer" containerID="4a1e90f7bd396ba5f2e98e2bc819e03d185f76a0b36b66b3282936ae6efa6599" Oct 30 13:50:38.898170 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:38.898147 5049 scope.go:122] "RemoveContainer" containerID="9691f7708a2f46f6d45527fbcbfc4fd10111d487f13b3617bea37a5cae454b71" Oct 30 13:50:38.908359 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:38.908238 5049 scope.go:122] "RemoveContainer" containerID="b230a5e8357cdd3a7d2f0c76ed92b80941a81fc9d0e0da1e45c5dbceab112b86" Oct 30 13:50:38.917269 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:38.917250 5049 scope.go:122] "RemoveContainer" containerID="985f44949db928439eaed7df39d65d2ae55ac6a556529f46f55799319511585d" Oct 30 13:50:38.925157 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:38.925132 5049 scope.go:122] "RemoveContainer" containerID="240bc13a69eff835d983881e8ff6804712ef7ef96e08e06acc0be3b310a72c0f" Oct 30 13:50:38.933363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:38.933316 5049 scope.go:122] "RemoveContainer" containerID="0d8f165e3f43c86fd26bc9ab20f0ca3f54c6d4366eb9b1e1e5687e5b11286f4c" Oct 30 13:50:38.941441 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:38.941421 5049 scope.go:122] "RemoveContainer" containerID="d0e0303e1b3ea25fb2675bed5e2c03628b48a7bdfad0e4e1b81adbb502c8940d" Oct 30 13:50:38.948984 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:38.948964 5049 scope.go:122] "RemoveContainer" containerID="a7b9994a25b9c274356dc91c8cbb925ff877b1c595a51fe5b859ca40a137781f" Oct 30 13:50:56.679962 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:50:56.679915 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:50:59.611159 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:50:59.611123 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:00.206750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:00.206710 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:00.381556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:00.381510 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:03.698102 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:03.698071 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:03.899686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:03.899651 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:05.287936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:05.287867 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:05.490999 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:05.490962 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:07.380196 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:07.380160 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:09.004390 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:09.001531 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:10.606122 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:10.606092 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:12.319354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:12.319305 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:13.937863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:13.937818 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:16.044158 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:16.044124 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:17.484676 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:17.484644 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:18.989974 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:18.989944 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:19.155125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:19.155093 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:19.722303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:19.722268 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:19.919860 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:19.919822 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:20.952090 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:20.952051 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:21.314194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:21.314155 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:22.640523 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:22.640475 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:24.526381 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:24.526297 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:26.136592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:26.136545 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:26.677813 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:51:26.677772 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:51:27.852298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:27.852219 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:29.409428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:29.409392 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:31.067228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:31.067149 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:31.390729 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:31.390614 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:32.879056 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:32.879004 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:34.114482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:34.114452 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:51:34.122674 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:34.122642 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:51:34.410540 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:34.410465 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:36.026190 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:36.026154 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:37.701083 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:37.701047 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:39.008807 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:39.008776 5049 scope.go:122] "RemoveContainer" containerID="5089b8aa008d9b308c62c9f8f6f3cc6adb35094f3e3f5b376cb1eea0e6be5e3c" Oct 30 13:51:39.019720 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:39.019679 5049 scope.go:122] "RemoveContainer" containerID="dd9c07d4b999eff663e24fba7c9c0434b874d027422642f7006c14fff305409e" Oct 30 13:51:39.029377 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:39.029348 5049 scope.go:122] "RemoveContainer" containerID="6c6ed1b221f96cfb4024c9b922beb87e051999deb842f7eb59f56bc721961c2b" Oct 30 13:51:39.037668 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:39.037646 5049 scope.go:122] "RemoveContainer" containerID="f971778356a646a3cf7cd39242ed6a2e90d2d38f24ae539cb9dab94dc9654871" Oct 30 13:51:39.047538 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:39.047433 5049 scope.go:122] "RemoveContainer" containerID="89a969315e29a1e7b61340562e886a09af6e183c9c520c5f5ef1a18d76e212a7" Oct 30 13:51:39.058038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:39.058015 5049 scope.go:122] "RemoveContainer" containerID="b4b934459abbed5c3e017f51e41a64934352d57587f8d36a1f77fca221ac2340" Oct 30 13:51:39.066128 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:39.066098 5049 scope.go:122] "RemoveContainer" containerID="54f27db614940c2f010462de057c05b5555547831213694dcecb398f5f9b30d6" Oct 30 13:51:39.074000 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:39.073978 5049 scope.go:122] "RemoveContainer" containerID="5d39c23654e2dddbad7bbfb2ad8c56eea45637399862187904af572dc16d5194" Oct 30 13:51:39.081712 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:39.081694 5049 scope.go:122] "RemoveContainer" containerID="16bc54f506752a7eefa44ebc5a4b2fc8c886252e388a09dad60a8ef9cd84ebcf" Oct 30 13:51:39.386574 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:39.386536 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:41.067177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:41.067145 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:42.582374 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:42.582341 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:44.311127 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:44.311096 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:45.715867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.715831 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj"] Oct 30 13:51:45.716457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.716435 5049 cpu_manager.go:414] "containerMap: removing container" podUID="9b998117-6bfa-410a-8e42-293cd4024c7a" containerName="revoke-keys" Oct 30 13:51:45.716457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.716456 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b998117-6bfa-410a-8e42-293cd4024c7a" containerName="revoke-keys" Oct 30 13:51:45.716645 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.716545 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b998117-6bfa-410a-8e42-293cd4024c7a" containerName="revoke-keys" Oct 30 13:51:45.721626 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.721606 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:45.734448 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.734422 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:51:45.744501 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.744467 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:51:45.744630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.744511 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:51:45.744630 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.744562 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:51:45.747213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.747188 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj"] Oct 30 13:51:45.756203 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.756178 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kmv2\" (UniqueName: \"kubernetes.io/projected/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-kube-api-access-6kmv2\") pod \"maas-api-revoke-sub-fc5ec89e-7mfmj\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:45.756353 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.756305 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc5ec89e-7mfmj\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:45.756436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.756392 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc5ec89e-7mfmj\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:45.856843 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.856809 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmv2\" (UniqueName: \"kubernetes.io/projected/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-kube-api-access-6kmv2\") pod \"maas-api-revoke-sub-fc5ec89e-7mfmj\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:45.857033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.856870 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc5ec89e-7mfmj\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:45.857033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.856909 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc5ec89e-7mfmj\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:45.857691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.857669 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc5ec89e-7mfmj\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:45.859435 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.859408 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc5ec89e-7mfmj\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:45.878096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.878070 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmv2\" (UniqueName: \"kubernetes.io/projected/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-kube-api-access-6kmv2\") pod \"maas-api-revoke-sub-fc5ec89e-7mfmj\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:45.891910 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:45.891876 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:46.035840 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:46.035760 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:46.335461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:46.335424 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj"] Oct 30 13:51:46.339638 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:51:46.339605 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdd9d39_d45b_4dd8_a0a1_8bd4c5ec0613.slice/crio-e50b88b501c3f8efefb00f7b1d588085805fdd7b6925188092cdcccab00f8655 WatchSource:0}: Error finding container e50b88b501c3f8efefb00f7b1d588085805fdd7b6925188092cdcccab00f8655: Status 404 returned error can't find the container with id e50b88b501c3f8efefb00f7b1d588085805fdd7b6925188092cdcccab00f8655 Oct 30 13:51:46.485474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:46.485438 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" event={"ID":"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613","Type":"ContainerStarted","Data":"e50b88b501c3f8efefb00f7b1d588085805fdd7b6925188092cdcccab00f8655"} Oct 30 13:51:47.470315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:47.470231 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:47.493245 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:47.493215 5049 generic.go:356] "Generic (PLEG): container finished" podID="ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613" containerID="0c3bfbd8d24fed285acf78c7ef31e973c478b8f8b23320c23dac5f1d0dad544f" exitCode=0 Oct 30 13:51:47.493428 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:47.493299 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" event={"ID":"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613","Type":"ContainerDied","Data":"0c3bfbd8d24fed285acf78c7ef31e973c478b8f8b23320c23dac5f1d0dad544f"} Oct 30 13:51:48.652744 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:48.652720 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:48.682131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:48.682104 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-xks-ca\") pod \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " Oct 30 13:51:48.682313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:48.682139 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-kube-api-access-6kmv2\" (UniqueName: \"kubernetes.io/projected/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-kube-api-access-6kmv2\") pod \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " Oct 30 13:51:48.682313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:48.682190 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-openshift-ca\") pod \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\" (UID: \"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613\") " Oct 30 13:51:48.682670 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:48.682637 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-openshift-ca" pod "ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613" (UID: "ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:51:48.684403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:48.684371 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-xks-ca" pod "ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613" (UID: "ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:51:48.684537 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:48.684461 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-kube-api-access-6kmv2" pod "ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613" (UID: "ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613"). InnerVolumeSpecName "kube-api-access-6kmv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:51:48.783554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:48.783474 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:51:48.783554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:48.783504 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kmv2\" (UniqueName: \"kubernetes.io/projected/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-kube-api-access-6kmv2\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:51:48.783554 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:48.783515 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:51:49.028446 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:49.028413 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:49.508659 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:49.508627 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" event={"ID":"ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613","Type":"ContainerDied","Data":"e50b88b501c3f8efefb00f7b1d588085805fdd7b6925188092cdcccab00f8655"} Oct 30 13:51:49.508863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:49.508687 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e50b88b501c3f8efefb00f7b1d588085805fdd7b6925188092cdcccab00f8655" Oct 30 13:51:49.508863 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:49.508635 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj" Oct 30 13:51:50.787422 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:50.787384 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:52.269006 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:52.268954 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:53.775133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:53.775090 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:55.495929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:55.495830 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:56.680493 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:51:56.680440 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:51:56.759457 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.759423 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:56.882681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.882644 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7"] Oct 30 13:51:56.883415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.883387 5049 cpu_manager.go:414] "containerMap: removing container" podUID="ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613" containerName="revoke-keys" Oct 30 13:51:56.883415 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.883413 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613" containerName="revoke-keys" Oct 30 13:51:56.883588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.883519 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613" containerName="revoke-keys" Oct 30 13:51:56.887358 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.887340 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:56.900208 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.900181 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:51:56.900387 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.900183 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:51:56.910616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.910587 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:51:56.910773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.910687 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:51:56.913223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.913195 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7"] Oct 30 13:51:56.961266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.961193 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac3c083-87f8-4889-b361-731e31062499-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:56.961266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.961261 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac3c083-87f8-4889-b361-731e31062499-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:56.961474 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:56.961366 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfwv\" (UniqueName: \"kubernetes.io/projected/8ac3c083-87f8-4889-b361-731e31062499-kube-api-access-nnfwv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:57.062918 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:57.062870 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac3c083-87f8-4889-b361-731e31062499-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:57.063136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:57.062943 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac3c083-87f8-4889-b361-731e31062499-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:57.063136 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:57.062975 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfwv\" (UniqueName: \"kubernetes.io/projected/8ac3c083-87f8-4889-b361-731e31062499-kube-api-access-nnfwv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:57.063649 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:57.063618 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac3c083-87f8-4889-b361-731e31062499-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:57.065420 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:57.065396 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac3c083-87f8-4889-b361-731e31062499-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:57.088228 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:57.088193 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfwv\" (UniqueName: \"kubernetes.io/projected/8ac3c083-87f8-4889-b361-731e31062499-kube-api-access-nnfwv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:57.127403 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:57.127314 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:57.200363 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:57.200307 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:57.506768 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:57.506679 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7"] Oct 30 13:51:57.511068 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:51:57.511035 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac3c083_87f8_4889_b361_731e31062499.slice/crio-2523a54ceef8bdb34b19989f87a9d9fb3d58bc2fd0ae0c01bc795bbe36e05d16 WatchSource:0}: Error finding container 2523a54ceef8bdb34b19989f87a9d9fb3d58bc2fd0ae0c01bc795bbe36e05d16: Status 404 returned error can't find the container with id 2523a54ceef8bdb34b19989f87a9d9fb3d58bc2fd0ae0c01bc795bbe36e05d16 Oct 30 13:51:57.566842 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:57.566810 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" event={"ID":"8ac3c083-87f8-4889-b361-731e31062499","Type":"ContainerStarted","Data":"2523a54ceef8bdb34b19989f87a9d9fb3d58bc2fd0ae0c01bc795bbe36e05d16"} Oct 30 13:51:58.574201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:58.574164 5049 generic.go:356] "Generic (PLEG): container finished" podID="8ac3c083-87f8-4889-b361-731e31062499" containerID="781c358c162cd6523f408e14b3def263647bf4eedeba18dfbdac7fc3d92206d8" exitCode=0 Oct 30 13:51:58.574691 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:58.574217 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" event={"ID":"8ac3c083-87f8-4889-b361-731e31062499","Type":"ContainerDied","Data":"781c358c162cd6523f408e14b3def263647bf4eedeba18dfbdac7fc3d92206d8"} Oct 30 13:51:58.773940 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:58.773902 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:59.721425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:59.721402 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:51:59.791847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:59.791785 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac3c083-87f8-4889-b361-731e31062499-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac3c083-87f8-4889-b361-731e31062499-maas-api-openshift-ca\") pod \"8ac3c083-87f8-4889-b361-731e31062499\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " Oct 30 13:51:59.791847 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:59.791848 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac3c083-87f8-4889-b361-731e31062499-kube-api-access-nnfwv\" (UniqueName: \"kubernetes.io/projected/8ac3c083-87f8-4889-b361-731e31062499-kube-api-access-nnfwv\") pod \"8ac3c083-87f8-4889-b361-731e31062499\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " Oct 30 13:51:59.792101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:59.791917 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac3c083-87f8-4889-b361-731e31062499-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac3c083-87f8-4889-b361-731e31062499-maas-api-xks-ca\") pod \"8ac3c083-87f8-4889-b361-731e31062499\" (UID: \"8ac3c083-87f8-4889-b361-731e31062499\") " Oct 30 13:51:59.792306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:59.792281 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac3c083-87f8-4889-b361-731e31062499-maas-api-openshift-ca" pod "8ac3c083-87f8-4889-b361-731e31062499" (UID: "8ac3c083-87f8-4889-b361-731e31062499"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:51:59.794126 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:59.794098 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac3c083-87f8-4889-b361-731e31062499-maas-api-xks-ca" pod "8ac3c083-87f8-4889-b361-731e31062499" (UID: "8ac3c083-87f8-4889-b361-731e31062499"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:51:59.794241 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:59.794107 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac3c083-87f8-4889-b361-731e31062499-kube-api-access-nnfwv" pod "8ac3c083-87f8-4889-b361-731e31062499" (UID: "8ac3c083-87f8-4889-b361-731e31062499"). InnerVolumeSpecName "kube-api-access-nnfwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:51:59.893726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:59.893648 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac3c083-87f8-4889-b361-731e31062499-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:51:59.893726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:59.893681 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac3c083-87f8-4889-b361-731e31062499-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:51:59.893726 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:51:59.893702 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnfwv\" (UniqueName: \"kubernetes.io/projected/8ac3c083-87f8-4889-b361-731e31062499-kube-api-access-nnfwv\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:52:00.566885 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:00.566852 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:00.585627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:00.585600 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" Oct 30 13:52:00.585779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:00.585636 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7" event={"ID":"8ac3c083-87f8-4889-b361-731e31062499","Type":"ContainerDied","Data":"2523a54ceef8bdb34b19989f87a9d9fb3d58bc2fd0ae0c01bc795bbe36e05d16"} Oct 30 13:52:00.585779 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:00.585681 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2523a54ceef8bdb34b19989f87a9d9fb3d58bc2fd0ae0c01bc795bbe36e05d16" Oct 30 13:52:01.804616 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:01.804582 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:02.063896 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:02.063797 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:03.749615 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:03.749580 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:05.272648 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:05.272610 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:06.810443 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:06.810404 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:07.435016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:07.434985 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:07.595575 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:07.595534 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:08.120845 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:08.120815 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:08.699461 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:08.699410 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:08.744927 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:08.744889 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7"] Oct 30 13:52:08.755855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:08.755824 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a5b3f0-v4zm7"] Oct 30 13:52:08.852977 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:08.852947 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:09.335027 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:09.334994 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac3c083-87f8-4889-b361-731e31062499" path="/var/lib/kubelet/pods/8ac3c083-87f8-4889-b361-731e31062499/volumes" Oct 30 13:52:09.379436 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:09.379398 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:15.328811 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:15.328730 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:15.485101 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:15.485067 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:15.725773 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:15.725695 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:26.676796 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:52:26.676759 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:52:31.533769 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:31.533732 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:34.030106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:34.030065 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz"] Oct 30 13:52:34.045147 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:34.045112 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5vjzz"] Oct 30 13:52:35.334948 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:35.334914 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c48ac8-b089-4f0c-8ba4-712cd843b70a" path="/var/lib/kubelet/pods/04c48ac8-b089-4f0c-8ba4-712cd843b70a/volumes" Oct 30 13:52:39.337481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:39.337443 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:40.465472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:40.465442 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:41.119556 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.119513 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87"] Oct 30 13:52:41.119826 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.119783 5049 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" podUID="dae44be0-4edf-40eb-9ec1-bc20e5a8de75" containerName="manager" containerID="cri-o://a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84" gracePeriod=10 Oct 30 13:52:41.378379 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.378298 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 13:52:41.497906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.497858 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-kube-api-access-4rvzg\" (UniqueName: \"kubernetes.io/projected/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-kube-api-access-4rvzg\") pod \"dae44be0-4edf-40eb-9ec1-bc20e5a8de75\" (UID: \"dae44be0-4edf-40eb-9ec1-bc20e5a8de75\") " Oct 30 13:52:41.497906 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.497904 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-extensions-socket-volume\") pod \"dae44be0-4edf-40eb-9ec1-bc20e5a8de75\" (UID: \"dae44be0-4edf-40eb-9ec1-bc20e5a8de75\") " Oct 30 13:52:41.498550 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.498331 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-extensions-socket-volume" pod "dae44be0-4edf-40eb-9ec1-bc20e5a8de75" (UID: "dae44be0-4edf-40eb-9ec1-bc20e5a8de75"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:52:41.500065 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.500041 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-kube-api-access-4rvzg" pod "dae44be0-4edf-40eb-9ec1-bc20e5a8de75" (UID: "dae44be0-4edf-40eb-9ec1-bc20e5a8de75"). InnerVolumeSpecName "kube-api-access-4rvzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:52:41.599059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.599024 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rvzg\" (UniqueName: \"kubernetes.io/projected/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-kube-api-access-4rvzg\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:52:41.599059 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.599054 5049 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dae44be0-4edf-40eb-9ec1-bc20e5a8de75-extensions-socket-volume\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:52:41.848475 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.848435 5049 generic.go:356] "Generic (PLEG): container finished" podID="dae44be0-4edf-40eb-9ec1-bc20e5a8de75" containerID="a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84" exitCode=0 Oct 30 13:52:41.848662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.848483 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" Oct 30 13:52:41.848662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.848513 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" event={"ID":"dae44be0-4edf-40eb-9ec1-bc20e5a8de75","Type":"ContainerDied","Data":"a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84"} Oct 30 13:52:41.848662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.848572 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87" event={"ID":"dae44be0-4edf-40eb-9ec1-bc20e5a8de75","Type":"ContainerDied","Data":"b5e6c27fa3cc1d38c5b551bdf9985c7bf0afba4ecf4431a5cedef2c2d32ebc72"} Oct 30 13:52:41.848662 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.848594 5049 scope.go:122] "RemoveContainer" containerID="a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84" Oct 30 13:52:41.859183 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.859164 5049 scope.go:122] "RemoveContainer" containerID="a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84" Oct 30 13:52:41.859490 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:52:41.859470 5049 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84\": container with ID starting with a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84 not found: ID does not exist" containerID="a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84" Oct 30 13:52:41.859547 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.859499 5049 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84"} err="failed to get container status \"a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84\": rpc error: code = NotFound desc = could not find container \"a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84\": container with ID starting with a7e89d761ef331734b03b0a34f96ff4da67cb96b9c77e4e4fdda880576cd9a84 not found: ID does not exist" Oct 30 13:52:41.895970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.895941 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87"] Oct 30 13:52:41.899062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:41.899043 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mlv87"] Oct 30 13:52:43.334686 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:43.334652 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae44be0-4edf-40eb-9ec1-bc20e5a8de75" path="/var/lib/kubelet/pods/dae44be0-4edf-40eb-9ec1-bc20e5a8de75/volumes" Oct 30 13:52:45.027907 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:45.027870 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz"] Oct 30 13:52:45.037704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:45.037674 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rbhlz"] Oct 30 13:52:45.334316 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:45.334280 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f226258-9728-494c-8d84-4876af9fd10b" path="/var/lib/kubelet/pods/3f226258-9728-494c-8d84-4876af9fd10b/volumes" Oct 30 13:52:45.726836 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:52:45.726698 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:56.675693 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:52:56.675658 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:53:00.538894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:00.538802 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:14.028247 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:14.028214 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn"] Oct 30 13:53:14.038137 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:14.038109 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ldftn"] Oct 30 13:53:15.334287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:15.334248 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9140a58-764e-4c41-b031-0f688e95aa26" path="/var/lib/kubelet/pods/f9140a58-764e-4c41-b031-0f688e95aa26/volumes" Oct 30 13:53:23.182593 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:23.182558 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:24.045308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:24.045273 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb"] Oct 30 13:53:24.054848 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:24.054819 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7jzlb"] Oct 30 13:53:25.334016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:25.333972 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f36c9d-7049-4473-93aa-2eec11c72caf" path="/var/lib/kubelet/pods/d6f36c9d-7049-4473-93aa-2eec11c72caf/volumes" Oct 30 13:53:26.676744 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:53:26.676705 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:53:31.646598 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:31.646568 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:39.150017 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:39.149985 5049 scope.go:122] "RemoveContainer" containerID="578c35c2646d91fd65334a792ab8b1bb0c58ec2a29fd4a0677fbdc2bd12b2fed" Oct 30 13:53:41.757303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:41.757267 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:42.144133 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.144096 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l"] Oct 30 13:53:42.144610 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.144591 5049 cpu_manager.go:414] "containerMap: removing container" podUID="8ac3c083-87f8-4889-b361-731e31062499" containerName="revoke-keys" Oct 30 13:53:42.144717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.144612 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac3c083-87f8-4889-b361-731e31062499" containerName="revoke-keys" Oct 30 13:53:42.144717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.144642 5049 cpu_manager.go:414] "containerMap: removing container" podUID="dae44be0-4edf-40eb-9ec1-bc20e5a8de75" containerName="manager" Oct 30 13:53:42.144717 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.144650 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae44be0-4edf-40eb-9ec1-bc20e5a8de75" containerName="manager" Oct 30 13:53:42.144892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.144788 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac3c083-87f8-4889-b361-731e31062499" containerName="revoke-keys" Oct 30 13:53:42.144892 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.144806 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae44be0-4edf-40eb-9ec1-bc20e5a8de75" containerName="manager" Oct 30 13:53:42.149416 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.149393 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" Oct 30 13:53:42.163207 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.163184 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x7hnm\"" Oct 30 13:53:42.175969 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.175940 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l"] Oct 30 13:53:42.263637 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.263595 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-whs2l\" (UID: \"d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" Oct 30 13:53:42.263804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.263727 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh47n\" (UniqueName: \"kubernetes.io/projected/d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8-kube-api-access-zh47n\") pod \"kuadrant-operator-controller-manager-57bb9bfff-whs2l\" (UID: \"d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" Oct 30 13:53:42.364177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.364137 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh47n\" (UniqueName: \"kubernetes.io/projected/d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8-kube-api-access-zh47n\") pod \"kuadrant-operator-controller-manager-57bb9bfff-whs2l\" (UID: \"d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" Oct 30 13:53:42.364395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.364217 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-whs2l\" (UID: \"d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" Oct 30 13:53:42.364590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.364570 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-whs2l\" (UID: \"d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" Oct 30 13:53:42.387153 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.387122 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh47n\" (UniqueName: \"kubernetes.io/projected/d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8-kube-api-access-zh47n\") pod \"kuadrant-operator-controller-manager-57bb9bfff-whs2l\" (UID: \"d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" Oct 30 13:53:42.462412 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.462302 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" Oct 30 13:53:42.820665 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:42.820632 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l"] Oct 30 13:53:42.825376 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:53:42.825343 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a7c2fd_5806_4adc_96aa_2f93ff6c53a8.slice/crio-32b741aee32194b00510ac35cf5fa259c6c2406a275762deb02570519efd997d WatchSource:0}: Error finding container 32b741aee32194b00510ac35cf5fa259c6c2406a275762deb02570519efd997d: Status 404 returned error can't find the container with id 32b741aee32194b00510ac35cf5fa259c6c2406a275762deb02570519efd997d Oct 30 13:53:43.184015 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:43.183913 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" event={"ID":"d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8","Type":"ContainerStarted","Data":"c8b16a9a93f0db82e301e2d260cbd70126bfd981a27991000f0761d2faf7934e"} Oct 30 13:53:43.184184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:43.184017 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" event={"ID":"d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8","Type":"ContainerStarted","Data":"32b741aee32194b00510ac35cf5fa259c6c2406a275762deb02570519efd997d"} Oct 30 13:53:43.184184 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:43.184058 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" Oct 30 13:53:43.230213 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:43.230163 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" podStartSLOduration=1.230150955 podStartE2EDuration="1.230150955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:43.229092057 +0000 UTC m=+3730.044286479" watchObservedRunningTime="2026-10-30 13:53:43.230150955 +0000 UTC m=+3730.045345377" Oct 30 13:53:46.035301 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:46.035210 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl"] Oct 30 13:53:46.045541 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:46.045509 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6dbvl"] Oct 30 13:53:47.334152 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:47.334117 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b54735c-2985-438d-b5d7-c169a21a6c56" path="/var/lib/kubelet/pods/8b54735c-2985-438d-b5d7-c169a21a6c56/volumes" Oct 30 13:53:54.192535 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:53:54.192502 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-whs2l" Oct 30 13:53:56.677781 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:53:56.677740 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:54:07.027349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:07.027303 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk"] Oct 30 13:54:07.037701 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:07.037673 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h5ssk"] Oct 30 13:54:07.333682 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:07.333651 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f929ecc6-4b59-4b20-96c6-8a69365eb789" path="/var/lib/kubelet/pods/f929ecc6-4b59-4b20-96c6-8a69365eb789/volumes" Oct 30 13:54:26.676640 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:54:26.676600 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:54:27.028970 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:27.028857 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj"] Oct 30 13:54:27.037025 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:27.036994 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9f8gj"] Oct 30 13:54:27.334295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:27.334264 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1a4838-d99a-41b6-a45c-900ecefe5583" path="/var/lib/kubelet/pods/ec1a4838-d99a-41b6-a45c-900ecefe5583/volumes" Oct 30 13:54:31.768001 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:31.767967 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:34.351820 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:34.351787 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:34.975007 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:34.974976 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:35.136884 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:35.136828 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:38.858367 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:38.858301 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:39.019530 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:39.019496 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:39.181038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:39.180958 5049 scope.go:122] "RemoveContainer" containerID="d796efe1309624abed24d429438229f6f41f8434f35c8164bd0ff12c0abfd625" Oct 30 13:54:39.190481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:39.190426 5049 scope.go:122] "RemoveContainer" containerID="b7accb1dd78db8d1c7ccff0edf976893b0cdf2e1ef7ef9436e07bdc5f9beba22" Oct 30 13:54:39.200484 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:39.200462 5049 scope.go:122] "RemoveContainer" containerID="d70b7669d315dcff7c46bbd4498ea68873eba6f38b1b613e8a7a2d074fc42d9f" Oct 30 13:54:40.328681 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:40.328618 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:40.483889 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:40.483849 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:40.653243 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:40.653145 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:42.197330 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:42.197266 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:44.011176 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:44.011137 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:45.027366 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:45.027302 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk"] Oct 30 13:54:45.037559 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:45.037522 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q5xrk"] Oct 30 13:54:45.334240 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:45.334196 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023c09b8-1dd6-412e-a2d9-2330236037bd" path="/var/lib/kubelet/pods/023c09b8-1dd6-412e-a2d9-2330236037bd/volumes" Oct 30 13:54:45.629425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:45.621021 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:47.245725 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:47.245688 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:49.002380 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:49.002307 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:50.660304 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:50.660271 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:52.183602 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:52.183567 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:53.795937 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:53.795899 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:53.951822 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:53.951789 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:54.396968 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:54.396940 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:54.556772 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:54.556735 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:54.789053 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:54.788959 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:55.708407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:55.708308 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:56.060262 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:56.060230 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:56.677646 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:54:56.677608 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:54:57.539016 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:57.538979 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:59.185221 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:54:59.185183 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:00.916588 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:00.916550 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:02.565239 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:02.565207 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:04.384105 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:04.384070 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:06.017771 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:06.017728 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:07.719306 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:07.719261 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:09.330263 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:09.330226 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:10.866658 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:10.866561 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:12.554770 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:12.554731 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:14.287878 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:14.287788 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:15.467492 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.467457 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n"] Oct 30 13:55:15.471081 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.471063 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.484315 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.484294 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:55:15.494829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.494803 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:55:15.494829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.494809 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:55:15.495064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.495038 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:55:15.496925 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.496902 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n"] Oct 30 13:55:15.524709 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.524676 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ebe2e92-tj49n\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.524855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.524725 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ebe2e92-tj49n\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.524929 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.524876 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8kvw\" (UniqueName: \"kubernetes.io/projected/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-kube-api-access-m8kvw\") pod \"maas-api-revoke-sub-4ebe2e92-tj49n\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.625754 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.625716 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ebe2e92-tj49n\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.625932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.625774 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ebe2e92-tj49n\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.625932 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.625868 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kvw\" (UniqueName: \"kubernetes.io/projected/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-kube-api-access-m8kvw\") pod \"maas-api-revoke-sub-4ebe2e92-tj49n\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.626426 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.626394 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ebe2e92-tj49n\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.628162 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.628138 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ebe2e92-tj49n\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.647424 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.647393 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8kvw\" (UniqueName: \"kubernetes.io/projected/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-kube-api-access-m8kvw\") pod \"maas-api-revoke-sub-4ebe2e92-tj49n\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.784454 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.784381 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:15.995108 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:15.995070 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:16.084174 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:16.084145 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n"] Oct 30 13:55:16.086662 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:55:16.086627 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22a7739_bdfc_4296_9db2_b3fcb5e0636c.slice/crio-729a825ff3bff3dd648115f3c65bd84c8312e80511b63e94add2a242de65294e WatchSource:0}: Error finding container 729a825ff3bff3dd648115f3c65bd84c8312e80511b63e94add2a242de65294e: Status 404 returned error can't find the container with id 729a825ff3bff3dd648115f3c65bd84c8312e80511b63e94add2a242de65294e Oct 30 13:55:16.731992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:16.731899 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" event={"ID":"f22a7739-bdfc-4296-9db2-b3fcb5e0636c","Type":"ContainerStarted","Data":"9b71f1c024f6f4711f13c78e76c16c316fe2fc86da040dc7961535d73b1bf940"} Oct 30 13:55:16.731992 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:16.731960 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" event={"ID":"f22a7739-bdfc-4296-9db2-b3fcb5e0636c","Type":"ContainerStarted","Data":"729a825ff3bff3dd648115f3c65bd84c8312e80511b63e94add2a242de65294e"} Oct 30 13:55:16.769350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:16.769272 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" podStartSLOduration=1.769258547 podStartE2EDuration="1.769258547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:16.768306557 +0000 UTC m=+3823.583500980" watchObservedRunningTime="2026-10-30 13:55:16.769258547 +0000 UTC m=+3823.584452970" Oct 30 13:55:17.741372 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:17.741300 5049 generic.go:356] "Generic (PLEG): container finished" podID="f22a7739-bdfc-4296-9db2-b3fcb5e0636c" containerID="9b71f1c024f6f4711f13c78e76c16c316fe2fc86da040dc7961535d73b1bf940" exitCode=0 Oct 30 13:55:17.741792 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:17.741382 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" event={"ID":"f22a7739-bdfc-4296-9db2-b3fcb5e0636c","Type":"ContainerDied","Data":"9b71f1c024f6f4711f13c78e76c16c316fe2fc86da040dc7961535d73b1bf940"} Oct 30 13:55:17.753295 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:17.753268 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:18.885298 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:18.885275 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:18.962038 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:18.962006 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-kube-api-access-m8kvw\" (UniqueName: \"kubernetes.io/projected/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-kube-api-access-m8kvw\") pod \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " Oct 30 13:55:18.962223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:18.962061 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-xks-ca\") pod \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " Oct 30 13:55:18.962223 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:18.962194 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-openshift-ca\") pod \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\" (UID: \"f22a7739-bdfc-4296-9db2-b3fcb5e0636c\") " Oct 30 13:55:18.962623 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:18.962592 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-openshift-ca" pod "f22a7739-bdfc-4296-9db2-b3fcb5e0636c" (UID: "f22a7739-bdfc-4296-9db2-b3fcb5e0636c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:18.964355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:18.964308 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-xks-ca" pod "f22a7739-bdfc-4296-9db2-b3fcb5e0636c" (UID: "f22a7739-bdfc-4296-9db2-b3fcb5e0636c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:18.964355 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:18.964312 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-kube-api-access-m8kvw" pod "f22a7739-bdfc-4296-9db2-b3fcb5e0636c" (UID: "f22a7739-bdfc-4296-9db2-b3fcb5e0636c"). InnerVolumeSpecName "kube-api-access-m8kvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:19.062988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:19.062955 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:55:19.062988 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:19.062985 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8kvw\" (UniqueName: \"kubernetes.io/projected/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-kube-api-access-m8kvw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:55:19.063212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:19.062998 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22a7739-bdfc-4296-9db2-b3fcb5e0636c-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:55:19.692604 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:19.692524 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:19.752471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:19.752440 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" Oct 30 13:55:19.752471 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:19.752455 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n" event={"ID":"f22a7739-bdfc-4296-9db2-b3fcb5e0636c","Type":"ContainerDied","Data":"729a825ff3bff3dd648115f3c65bd84c8312e80511b63e94add2a242de65294e"} Oct 30 13:55:19.752678 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:19.752499 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="729a825ff3bff3dd648115f3c65bd84c8312e80511b63e94add2a242de65294e" Oct 30 13:55:21.250539 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:21.250497 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:22.873643 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:22.873610 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:24.446594 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:24.446561 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:26.021084 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.021044 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:26.378219 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.378184 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:26.469625 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.469591 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f"] Oct 30 13:55:26.470118 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.470097 5049 cpu_manager.go:414] "containerMap: removing container" podUID="f22a7739-bdfc-4296-9db2-b3fcb5e0636c" containerName="revoke-keys" Oct 30 13:55:26.470212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.470120 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22a7739-bdfc-4296-9db2-b3fcb5e0636c" containerName="revoke-keys" Oct 30 13:55:26.470285 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.470218 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22a7739-bdfc-4296-9db2-b3fcb5e0636c" containerName="revoke-keys" Oct 30 13:55:26.473612 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.473589 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:26.487029 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.487008 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:55:26.487272 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.487252 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:55:26.497857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.497711 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:55:26.497857 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.497824 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:55:26.500395 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.500369 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f"] Oct 30 13:55:26.635505 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.635412 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:26.635704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.635538 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:26.635704 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.635624 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722lw\" (UniqueName: \"kubernetes.io/projected/cdb2d525-8322-41c2-8198-267ce9ef9032-kube-api-access-722lw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:26.677134 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:55:26.677098 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:55:26.736790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.736750 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:26.736790 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.736791 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722lw\" (UniqueName: \"kubernetes.io/projected/cdb2d525-8322-41c2-8198-267ce9ef9032-kube-api-access-722lw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:26.737066 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.736901 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:26.737760 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.737735 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:26.739313 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.739285 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:26.758865 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.758829 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-722lw\" (UniqueName: \"kubernetes.io/projected/cdb2d525-8322-41c2-8198-267ce9ef9032-kube-api-access-722lw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:26.788981 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:26.788941 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:27.100687 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:27.098195 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f"] Oct 30 13:55:27.100687 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:55:27.099987 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb2d525_8322_41c2_8198_267ce9ef9032.slice/crio-9e2a18c9edc93dfcb48553479634cd58d25a076c0bd7e42cc0e7a994bd100ecd WatchSource:0}: Error finding container 9e2a18c9edc93dfcb48553479634cd58d25a076c0bd7e42cc0e7a994bd100ecd: Status 404 returned error can't find the container with id 9e2a18c9edc93dfcb48553479634cd58d25a076c0bd7e42cc0e7a994bd100ecd Oct 30 13:55:27.797125 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:27.797075 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" event={"ID":"cdb2d525-8322-41c2-8198-267ce9ef9032","Type":"ContainerStarted","Data":"941fbf46ddb6d9007b96c9b963f205faf5265ac6d5f5b62b638cfcde84979fc3"} Oct 30 13:55:27.797307 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:27.797144 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" event={"ID":"cdb2d525-8322-41c2-8198-267ce9ef9032","Type":"ContainerStarted","Data":"9e2a18c9edc93dfcb48553479634cd58d25a076c0bd7e42cc0e7a994bd100ecd"} Oct 30 13:55:27.835043 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:27.834981 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" podStartSLOduration=1.834967418 podStartE2EDuration="1.834967418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:27.833645221 +0000 UTC m=+3834.648839644" watchObservedRunningTime="2026-10-30 13:55:27.834967418 +0000 UTC m=+3834.650161842" Oct 30 13:55:27.890345 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:27.890275 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:28.804039 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:28.804002 5049 generic.go:356] "Generic (PLEG): container finished" podID="cdb2d525-8322-41c2-8198-267ce9ef9032" containerID="941fbf46ddb6d9007b96c9b963f205faf5265ac6d5f5b62b638cfcde84979fc3" exitCode=0 Oct 30 13:55:28.804481 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:28.804065 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" event={"ID":"cdb2d525-8322-41c2-8198-267ce9ef9032","Type":"ContainerDied","Data":"941fbf46ddb6d9007b96c9b963f205faf5265ac6d5f5b62b638cfcde84979fc3"} Oct 30 13:55:29.571734 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:29.571702 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:29.948794 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:29.948767 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:30.071104 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.071071 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-openshift-ca\") pod \"cdb2d525-8322-41c2-8198-267ce9ef9032\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " Oct 30 13:55:30.071266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.071129 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb2d525-8322-41c2-8198-267ce9ef9032-kube-api-access-722lw\" (UniqueName: \"kubernetes.io/projected/cdb2d525-8322-41c2-8198-267ce9ef9032-kube-api-access-722lw\") pod \"cdb2d525-8322-41c2-8198-267ce9ef9032\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " Oct 30 13:55:30.071266 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.071209 5049 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-xks-ca\") pod \"cdb2d525-8322-41c2-8198-267ce9ef9032\" (UID: \"cdb2d525-8322-41c2-8198-267ce9ef9032\") " Oct 30 13:55:30.071552 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.071522 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-openshift-ca" pod "cdb2d525-8322-41c2-8198-267ce9ef9032" (UID: "cdb2d525-8322-41c2-8198-267ce9ef9032"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:30.071695 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.071671 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-openshift-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:55:30.073400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.073375 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb2d525-8322-41c2-8198-267ce9ef9032-kube-api-access-722lw" pod "cdb2d525-8322-41c2-8198-267ce9ef9032" (UID: "cdb2d525-8322-41c2-8198-267ce9ef9032"). InnerVolumeSpecName "kube-api-access-722lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:30.073563 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.073539 5049 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-xks-ca" pod "cdb2d525-8322-41c2-8198-267ce9ef9032" (UID: "cdb2d525-8322-41c2-8198-267ce9ef9032"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:30.172941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.172810 5049 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722lw\" (UniqueName: \"kubernetes.io/projected/cdb2d525-8322-41c2-8198-267ce9ef9032-kube-api-access-722lw\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:55:30.172941 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.172842 5049 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb2d525-8322-41c2-8198-267ce9ef9032-maas-api-xks-ca\") on node \"ip-10-0-133-61.ec2.internal\" DevicePath \"\"" Oct 30 13:55:30.823352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.818279 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" event={"ID":"cdb2d525-8322-41c2-8198-267ce9ef9032","Type":"ContainerDied","Data":"9e2a18c9edc93dfcb48553479634cd58d25a076c0bd7e42cc0e7a994bd100ecd"} Oct 30 13:55:30.823352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.818410 5049 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e2a18c9edc93dfcb48553479634cd58d25a076c0bd7e42cc0e7a994bd100ecd" Oct 30 13:55:30.823352 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:30.818499 5049 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f" Oct 30 13:55:31.403891 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:31.403853 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:31.564037 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:31.564001 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:31.896202 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:31.896170 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:32.975033 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:32.974998 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:34.684671 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:34.684610 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:36.291201 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:36.291168 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:37.029194 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:37.029161 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:37.244096 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:37.244062 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:37.738036 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:37.738000 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:38.248837 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:38.248788 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:38.327713 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:38.327679 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f"] Oct 30 13:55:38.338035 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:38.338003 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d61c7a-tnn8f"] Oct 30 13:55:38.398060 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:38.398030 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:38.884979 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:38.884945 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:39.231463 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:39.231377 5049 scope.go:122] "RemoveContainer" containerID="2cadb7e9c4446f370e1fc5c9fa557eede556b1136015301a6ab1c805ccb6a556" Oct 30 13:55:39.240829 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:39.240786 5049 scope.go:122] "RemoveContainer" containerID="01cb1049b2df3cad2cbe8bf9a7659b75f1c9e1fa00213ba3b0b6f571aab9af5b" Oct 30 13:55:39.249110 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:39.249087 5049 scope.go:122] "RemoveContainer" containerID="2628813fba6c842b60c97751fb11cc85bc7297c5656210ee2039d1c67cfa2459" Oct 30 13:55:39.334805 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:39.334775 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb2d525-8322-41c2-8198-267ce9ef9032" path="/var/lib/kubelet/pods/cdb2d525-8322-41c2-8198-267ce9ef9032/volumes" Oct 30 13:55:45.543106 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:45.543071 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:45.711317 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:45.711285 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:45.946009 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:45.945925 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:50.008849 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:55:50.008811 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:56.683861 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:55:56.683821 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:56:07.144425 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:07.144384 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:08.819747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:08.819704 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:09.058590 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:09.058556 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:24.301795 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:24.301759 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:26.681806 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:56:26.681771 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:56:32.082348 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:32.081529 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:32.257764 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:32.257708 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:34.149407 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:34.149373 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:56:34.158577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:34.158550 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 13:56:39.281627 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:39.281590 5049 scope.go:122] "RemoveContainer" containerID="149d0b420866976350aca9a21f4d961f7154ad387487d7715c503f455bf2df18" Oct 30 13:56:39.290087 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:39.290065 5049 scope.go:122] "RemoveContainer" containerID="29a3e4f5c74b7ba7ada2fb880936f90442947d900f21e60e932f469c5ecdb1a2" Oct 30 13:56:40.673497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:40.673465 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:40.828308 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:40.828248 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:42.485548 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:42.485507 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:48.284455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:48.284375 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:49.029894 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:49.029857 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj"] Oct 30 13:56:49.040354 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:49.040310 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc5ec89e-7mfmj"] Oct 30 13:56:49.334577 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:56:49.334542 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613" path="/var/lib/kubelet/pods/ecdd9d39-d45b-4dd8-a0a1-8bd4c5ec0613/volumes" Oct 30 13:56:56.676004 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:56:56.675967 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:57:03.162472 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:03.162439 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:26.677771 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:57:26.677732 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:57:31.926337 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.926293 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-w57x7"] Oct 30 13:57:31.926756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.926730 5049 cpu_manager.go:414] "containerMap: removing container" podUID="cdb2d525-8322-41c2-8198-267ce9ef9032" containerName="revoke-keys" Oct 30 13:57:31.926756 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.926743 5049 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb2d525-8322-41c2-8198-267ce9ef9032" containerName="revoke-keys" Oct 30 13:57:31.926855 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.926845 5049 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb2d525-8322-41c2-8198-267ce9ef9032" containerName="revoke-keys" Oct 30 13:57:31.930402 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.930385 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:31.959499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.959469 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:57:31.959499 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.959470 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 13:57:31.959672 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.959512 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zw9vx\"" Oct 30 13:57:31.965400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.965383 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:57:31.965400 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.965386 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 13:57:31.979089 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:31.979067 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-w57x7"] Oct 30 13:57:32.025151 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.025119 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083f466e-9720-4e5f-b2e9-1212cdc8d37e-maas-api-tls\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.025349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.025167 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lk56\" (UniqueName: \"kubernetes.io/projected/083f466e-9720-4e5f-b2e9-1212cdc8d37e-kube-api-access-5lk56\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.025349 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.025192 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083f466e-9720-4e5f-b2e9-1212cdc8d37e-openshift-service-ca\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.025455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.025400 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083f466e-9720-4e5f-b2e9-1212cdc8d37e-metrics-certs\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.126293 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.126251 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083f466e-9720-4e5f-b2e9-1212cdc8d37e-maas-api-tls\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.126482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.126337 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lk56\" (UniqueName: \"kubernetes.io/projected/083f466e-9720-4e5f-b2e9-1212cdc8d37e-kube-api-access-5lk56\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.126482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.126461 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083f466e-9720-4e5f-b2e9-1212cdc8d37e-openshift-service-ca\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.126579 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.126548 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083f466e-9720-4e5f-b2e9-1212cdc8d37e-metrics-certs\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.127165 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.127142 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083f466e-9720-4e5f-b2e9-1212cdc8d37e-openshift-service-ca\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.128804 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.128782 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083f466e-9720-4e5f-b2e9-1212cdc8d37e-maas-api-tls\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.129141 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.129122 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083f466e-9720-4e5f-b2e9-1212cdc8d37e-metrics-certs\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.149859 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.149832 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lk56\" (UniqueName: \"kubernetes.io/projected/083f466e-9720-4e5f-b2e9-1212cdc8d37e-kube-api-access-5lk56\") pod \"maas-api-595cc648c-w57x7\" (UID: \"083f466e-9720-4e5f-b2e9-1212cdc8d37e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.243460 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.243381 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:32.276217 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.276189 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:32.551741 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.551714 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-w57x7"] Oct 30 13:57:32.553869 ip-10-0-133-61 kubenswrapper[5049]: W1030 13:57:32.553846 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083f466e_9720_4e5f_b2e9_1212cdc8d37e.slice/crio-b9c8ad8ef8e5cdccab1164f1fad6f9d7b5203fbdc371c47ec7853623f736ba19 WatchSource:0}: Error finding container b9c8ad8ef8e5cdccab1164f1fad6f9d7b5203fbdc371c47ec7853623f736ba19: Status 404 returned error can't find the container with id b9c8ad8ef8e5cdccab1164f1fad6f9d7b5203fbdc371c47ec7853623f736ba19 Oct 30 13:57:32.555171 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.555144 5049 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:57:32.582138 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:32.582109 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" event={"ID":"083f466e-9720-4e5f-b2e9-1212cdc8d37e","Type":"ContainerStarted","Data":"b9c8ad8ef8e5cdccab1164f1fad6f9d7b5203fbdc371c47ec7853623f736ba19"} Oct 30 13:57:33.589897 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:33.589860 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" event={"ID":"083f466e-9720-4e5f-b2e9-1212cdc8d37e","Type":"ContainerStarted","Data":"82576f05d210bfff495a2bbafa5e993844b94f866d6b3e624513080ce35a351b"} Oct 30 13:57:33.590339 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:33.589929 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:39.603413 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:39.603372 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" Oct 30 13:57:39.657482 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:39.657424 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-w57x7" podStartSLOduration=8.296400565 podStartE2EDuration="8.657411255s" totalImagesPullingTime="361.01069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:57:33.640963192 +0000 UTC m=+3960.456157834" watchObservedRunningTime="2026-10-30 13:57:39.657411255 +0000 UTC m=+3966.472605678" Oct 30 13:57:39.694949 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:39.694907 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:39.923917 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:57:39.923817 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:56.677106 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:57:56.677063 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:58:21.000255 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:21.000173 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754459558f-vrhvp_152af948-ed7c-44ff-b10f-40bc2030b17f/authorino/0.log" Oct 30 13:58:26.676718 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:58:26.676679 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:58:31.615887 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:31.615857 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-d5f86698f-jvzrj_7b3822a7-c018-4112-a352-f436ce4b2de9/manager/0.log" Oct 30 13:58:32.441455 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:32.441414 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:32.748273 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:32.748182 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b844f68b7-qjmh7_f7f8e4f9-5a9f-43b2-957b-16e0a8cde956/manager/0.log" Oct 30 13:58:34.204303 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:34.204279 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-s2d8q_d4a5b501-d88a-4063-9009-e59d5d0d0fe9/copy-manifests/0.log" Oct 30 13:58:34.244632 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:34.244598 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-s2d8q_d4a5b501-d88a-4063-9009-e59d5d0d0fe9/manager/0.log" Oct 30 13:58:34.461350 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:34.461259 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-2kk5k_8d3d062c-dc7f-4625-94ad-bb73e9c5244b/proxy/0.log" Oct 30 13:58:35.438227 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:35.438193 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-w57x7_083f466e-9720-4e5f-b2e9-1212cdc8d37e/maas-api/0.log" Oct 30 13:58:36.213747 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:36.213717 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4ebe2e92-tj49n_f22a7739-bdfc-4296-9db2-b3fcb5e0636c/revoke-keys/0.log" Oct 30 13:58:38.843586 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:38.843553 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754459558f-vrhvp_152af948-ed7c-44ff-b10f-40bc2030b17f/authorino/0.log" Oct 30 13:58:39.102867 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:39.102789 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-q6vtv_89cc7744-e78f-4659-a285-5d71985c6c12/manager/0.log" Oct 30 13:58:39.313557 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:39.313524 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-nddfq_0d345919-e44b-4028-80ee-f748cd3a53de/manager/0.log" Oct 30 13:58:39.322131 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:39.322104 5049 scope.go:122] "RemoveContainer" containerID="0c3bfbd8d24fed285acf78c7ef31e973c478b8f8b23320c23dac5f1d0dad544f" Oct 30 13:58:39.332064 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:39.332033 5049 scope.go:122] "RemoveContainer" containerID="781c358c162cd6523f408e14b3def263647bf4eedeba18dfbdac7fc3d92206d8" Oct 30 13:58:39.702361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:39.702270 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-whs2l_d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8/manager/0.log" Oct 30 13:58:40.140497 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:40.140465 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-mv4d8_ac4d89bd-42a7-4527-8838-5fdf95fd54ee/manager/0.log" Oct 30 13:58:41.947181 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:41.947150 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-5tb64_e381cf09-b412-436c-a024-26f22743c7c6/istio-proxy/0.log" Oct 30 13:58:42.171361 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:42.171331 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-b8pzq_deb02668-044a-472c-be76-77c2cf53c3f9/istio-proxy/0.log" Oct 30 13:58:42.365486 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:42.365452 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-jfdbw_da100cec-8aca-4cba-b19d-4c873acb6f2a/istio-proxy/0.log" Oct 30 13:58:43.807592 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:43.807561 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9mvqm_a1ace317-daee-47d5-a744-10547abad774/payload-pre-processing/0.log" Oct 30 13:58:44.210750 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:44.210670 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-df858cf77-mnl8f_173ce15b-35dd-4a98-8f29-d13c62b56206/router/0.log" Oct 30 13:58:45.416069 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:45.416036 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-lmr2v_a53b4e96-a86d-4f7d-8fe9-727c0bb79b23/main/0.log" Oct 30 13:58:45.803851 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:45.803821 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh_579177ce-7692-4a1b-be47-09b365d67e25/main/0.log" Oct 30 13:58:46.007200 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:46.007171 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk_9331bf25-1c21-420d-b7cf-fb931432db23/main/0.log" Oct 30 13:58:46.202858 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:46.202786 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6_dad6d45c-f49c-490c-906a-94b25d2b9430/main/0.log" Oct 30 13:58:47.137085 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:47.137051 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754459558f-vrhvp_152af948-ed7c-44ff-b10f-40bc2030b17f/authorino/0.log" Oct 30 13:58:56.385587 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:56.385554 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-d5f86698f-jvzrj_7b3822a7-c018-4112-a352-f436ce4b2de9/manager/0.log" Oct 30 13:58:56.676592 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:58:56.676500 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:58:57.336279 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:57.336253 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b844f68b7-qjmh7_f7f8e4f9-5a9f-43b2-957b-16e0a8cde956/manager/0.log" Oct 30 13:58:58.786655 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:58.786624 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-s2d8q_d4a5b501-d88a-4063-9009-e59d5d0d0fe9/copy-manifests/0.log" Oct 30 13:58:58.825054 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:58.825027 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-s2d8q_d4a5b501-d88a-4063-9009-e59d5d0d0fe9/manager/0.log" Oct 30 13:58:59.057364 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:59.057332 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-2kk5k_8d3d062c-dc7f-4625-94ad-bb73e9c5244b/proxy/0.log" Oct 30 13:58:59.817026 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:58:59.816999 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-w57x7_083f466e-9720-4e5f-b2e9-1212cdc8d37e/maas-api/0.log" Oct 30 13:59:00.581246 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:00.581210 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4ebe2e92-tj49n_f22a7739-bdfc-4296-9db2-b3fcb5e0636c/revoke-keys/0.log" Oct 30 13:59:03.020287 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:03.020255 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754459558f-vrhvp_152af948-ed7c-44ff-b10f-40bc2030b17f/authorino/0.log" Oct 30 13:59:03.286294 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:03.286214 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-q6vtv_89cc7744-e78f-4659-a285-5d71985c6c12/manager/0.log" Oct 30 13:59:03.484205 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:03.484174 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-nddfq_0d345919-e44b-4028-80ee-f748cd3a53de/manager/0.log" Oct 30 13:59:03.896168 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:03.896137 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-whs2l_d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8/manager/0.log" Oct 30 13:59:04.318062 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:04.318029 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-mv4d8_ac4d89bd-42a7-4527-8838-5fdf95fd54ee/manager/0.log" Oct 30 13:59:05.948603 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:05.948555 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-5tb64_e381cf09-b412-436c-a024-26f22743c7c6/istio-proxy/0.log" Oct 30 13:59:06.156743 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:06.156709 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-b8pzq_deb02668-044a-472c-be76-77c2cf53c3f9/istio-proxy/0.log" Oct 30 13:59:06.378311 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:06.378283 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-jfdbw_da100cec-8aca-4cba-b19d-4c873acb6f2a/istio-proxy/0.log" Oct 30 13:59:07.811987 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:07.811955 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9mvqm_a1ace317-daee-47d5-a744-10547abad774/payload-pre-processing/0.log" Oct 30 13:59:08.240332 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:08.240220 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-df858cf77-mnl8f_173ce15b-35dd-4a98-8f29-d13c62b56206/router/0.log" Oct 30 13:59:09.227936 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:09.227877 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-lmr2v_a53b4e96-a86d-4f7d-8fe9-727c0bb79b23/main/0.log" Oct 30 13:59:09.626230 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:09.626200 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8qwh_579177ce-7692-4a1b-be47-09b365d67e25/main/0.log" Oct 30 13:59:09.840177 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:09.840151 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2pwmk_9331bf25-1c21-420d-b7cf-fb931432db23/main/0.log" Oct 30 13:59:10.040214 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:10.040137 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-td5k6_dad6d45c-f49c-490c-906a-94b25d2b9430/main/0.log" Oct 30 13:59:26.676130 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:59:26.676090 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 13:59:32.412212 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:32.412169 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:46.172846 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:46.172762 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b844f68b7-qjmh7_f7f8e4f9-5a9f-43b2-957b-16e0a8cde956/manager/0.log" Oct 30 13:59:46.474707 ip-10-0-133-61 kubenswrapper[5049]: I1030 13:59:46.474627 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b844f68b7-qjmh7_f7f8e4f9-5a9f-43b2-957b-16e0a8cde956/manager/0.log" Oct 30 13:59:56.676232 ip-10-0-133-61 kubenswrapper[5049]: E1030 13:59:56.676193 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 14:00:19.029393 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:19.029349 5049 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n"] Oct 30 14:00:19.039392 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:19.039363 5049 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ebe2e92-tj49n"] Oct 30 14:00:19.333989 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:19.333959 5049 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22a7739-bdfc-4296-9db2-b3fcb5e0636c" path="/var/lib/kubelet/pods/f22a7739-bdfc-4296-9db2-b3fcb5e0636c/volumes" Oct 30 14:00:26.676519 ip-10-0-133-61 kubenswrapper[5049]: E1030 14:00:26.676483 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 14:00:32.376380 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:32.376346 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b844f68b7-qjmh7_f7f8e4f9-5a9f-43b2-957b-16e0a8cde956/manager/0.log" Oct 30 14:00:32.589583 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:32.589548 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:32.658670 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:32.658597 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b844f68b7-qjmh7_f7f8e4f9-5a9f-43b2-957b-16e0a8cde956/manager/0.log" Oct 30 14:00:33.950376 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:33.950309 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:34.129657 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:34.129622 5049 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:38.667257 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:38.667206 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jtc6z_00f1e4e7-b0ec-4c54-8429-1dd5ba02b0b1/global-pull-secret-syncer/0.log" Oct 30 14:00:39.081363 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:39.081332 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-97qv2_35925210-93d9-4a34-b733-6481da8921f3/connection-checker/0.log" Oct 30 14:00:39.216907 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:39.216879 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bqzs4_3400463b-e945-48d9-8dc5-8b8106cb6e26/connection-checker/0.log" Oct 30 14:00:39.351198 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:39.351121 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-crmm8_cbb88cf5-e088-4260-82e6-ed56a2b00176/connection-checker/0.log" Oct 30 14:00:39.524254 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:39.524211 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cdptr_b0aa96f3-47c7-48f1-98a3-dd3c4548d593/konnectivity-agent/0.log" Oct 30 14:00:40.004556 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:40.004522 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-61.ec2.internal_bac4afb01d6e1d9bf4827fcda4b8b291/haproxy/0.log" Oct 30 14:00:46.327023 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:46.326988 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754459558f-vrhvp_152af948-ed7c-44ff-b10f-40bc2030b17f/authorino/0.log" Oct 30 14:00:46.435386 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:46.435314 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-q6vtv_89cc7744-e78f-4659-a285-5d71985c6c12/manager/0.log" Oct 30 14:00:46.567042 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:46.566969 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-nddfq_0d345919-e44b-4028-80ee-f748cd3a53de/manager/0.log" Oct 30 14:00:46.853729 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:46.853694 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-whs2l_d6a7c2fd-5806-4adc-96aa-2f93ff6c53a8/manager/0.log" Oct 30 14:00:47.147548 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.147462 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-mv4d8_ac4d89bd-42a7-4527-8838-5fdf95fd54ee/manager/0.log" Oct 30 14:00:47.775851 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.775805 5049 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57"] Oct 30 14:00:47.779618 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.779590 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:47.794781 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.794759 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rkb8w\"/\"openshift-service-ca.crt\"" Oct 30 14:00:47.805434 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.805407 5049 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rkb8w\"/\"default-dockercfg-dpzm8\"" Oct 30 14:00:47.807390 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.807369 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57"] Oct 30 14:00:47.894593 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.894561 5049 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rkb8w\"/\"kube-root-ca.crt\"" Oct 30 14:00:47.942037 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.941999 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-podres\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:47.942227 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.942078 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n82m\" (UniqueName: \"kubernetes.io/projected/13e56d78-5a70-4555-a545-ccd93a5cec45-kube-api-access-7n82m\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:47.942227 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.942119 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-lib-modules\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:47.942227 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.942138 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-proc\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:47.942456 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:47.942248 5049 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-sys\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.043004 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.042917 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-podres\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.043004 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.042983 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n82m\" (UniqueName: \"kubernetes.io/projected/13e56d78-5a70-4555-a545-ccd93a5cec45-kube-api-access-7n82m\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.043234 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.043023 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-lib-modules\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.043234 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.043045 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-proc\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.043234 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.043059 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-podres\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.043234 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.043081 5049 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-sys\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.043234 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.043125 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-proc\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.043234 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.043156 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-lib-modules\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.043234 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.043177 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13e56d78-5a70-4555-a545-ccd93a5cec45-sys\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.064567 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.064534 5049 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n82m\" (UniqueName: \"kubernetes.io/projected/13e56d78-5a70-4555-a545-ccd93a5cec45-kube-api-access-7n82m\") pod \"perf-node-gather-daemonset-c7b57\" (UID: \"13e56d78-5a70-4555-a545-ccd93a5cec45\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.092433 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.092408 5049 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.389787 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.389757 5049 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57"] Oct 30 14:00:48.391996 ip-10-0-133-61 kubenswrapper[5049]: W1030 14:00:48.391967 5049 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod13e56d78_5a70_4555_a545_ccd93a5cec45.slice/crio-b12dc0693e623a209123a8dc21d2dd795fef4da79d0f53606988bbe78dcb3c2f WatchSource:0}: Error finding container b12dc0693e623a209123a8dc21d2dd795fef4da79d0f53606988bbe78dcb3c2f: Status 404 returned error can't find the container with id b12dc0693e623a209123a8dc21d2dd795fef4da79d0f53606988bbe78dcb3c2f Oct 30 14:00:48.688686 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.688596 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" event={"ID":"13e56d78-5a70-4555-a545-ccd93a5cec45","Type":"ContainerStarted","Data":"0a2b6eeb72a54dee6ea8106636f8a8ee041ec3f5dfc2bb627728b67b01c7ecef"} Oct 30 14:00:48.688686 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.688661 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:48.688686 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.688680 5049 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" event={"ID":"13e56d78-5a70-4555-a545-ccd93a5cec45","Type":"ContainerStarted","Data":"b12dc0693e623a209123a8dc21d2dd795fef4da79d0f53606988bbe78dcb3c2f"} Oct 30 14:00:48.727569 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:48.727520 5049 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" podStartSLOduration=1.7275070590000001 podStartE2EDuration="1.727507059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:48.726871325 +0000 UTC m=+4155.542065748" watchObservedRunningTime="2026-10-30 14:00:48.727507059 +0000 UTC m=+4155.542701482" Oct 30 14:00:49.657255 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:49.657223 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-jlskm_fcd977c1-bd5b-466b-a9bd-aa4889ed86b4/cluster-monitoring-operator/0.log" Oct 30 14:00:50.824720 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:50.824684 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tb2dm_2b483a39-dd4d-4049-ac91-8d566b1c01ab/node-exporter/0.log" Oct 30 14:00:50.950839 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:50.950804 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tb2dm_2b483a39-dd4d-4049-ac91-8d566b1c01ab/kube-rbac-proxy/0.log" Oct 30 14:00:51.078478 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:51.078406 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tb2dm_2b483a39-dd4d-4049-ac91-8d566b1c01ab/init-textfile/0.log" Oct 30 14:00:54.704463 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:54.704435 5049 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-c7b57" Oct 30 14:00:56.178719 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:56.178682 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-f4hf7_caed0908-537e-4064-858e-f3f8a82b87b1/networking-console-plugin/0.log" Oct 30 14:00:56.676368 ip-10-0-133-61 kubenswrapper[5049]: E1030 14:00:56.676310 5049 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bzlvd" containerName="extractor" Oct 30 14:00:57.429466 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:57.429431 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/1.log" Oct 30 14:00:57.433291 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:00:57.433270 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2xkr_264f3a6d-7988-4ee8-bf15-bf13e991a63c/console-operator/2.log" Oct 30 14:01:00.055524 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:01:00.055493 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-vf8sw_9dc9e5e5-426c-4e0b-897f-54e0e28d3313/volume-data-source-validator/0.log" Oct 30 14:01:02.465819 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:01:02.465790 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94rws_116675da-08c0-4101-835c-48193661ccbf/dns/0.log" Oct 30 14:01:02.595692 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:01:02.595661 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94rws_116675da-08c0-4101-835c-48193661ccbf/kube-rbac-proxy/0.log" Oct 30 14:01:03.524638 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:01:03.524600 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v4l8n_d0471989-f151-47a5-9d39-97d85760da9e/dns-node-resolver/0.log" Oct 30 14:01:04.478963 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:01:04.478931 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-5tb64_e381cf09-b412-436c-a024-26f22743c7c6/istio-proxy/0.log" Oct 30 14:01:04.544226 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:01:04.544191 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-b8pzq_deb02668-044a-472c-be76-77c2cf53c3f9/istio-proxy/0.log" Oct 30 14:01:04.604196 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:01:04.604158 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-jfdbw_da100cec-8aca-4cba-b19d-4c873acb6f2a/istio-proxy/0.log" Oct 30 14:01:04.817468 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:01:04.817441 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6p76v_126669c1-ee47-49ab-a6f3-3592a66b1fd2/node-ca/0.log" Oct 30 14:01:05.055503 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:01:05.055468 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9mvqm_a1ace317-daee-47d5-a744-10547abad774/payload-pre-processing/0.log" Oct 30 14:01:05.181919 ip-10-0-133-61 kubenswrapper[5049]: I1030 14:01:05.181844 5049 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-df858cf77-mnl8f_173ce15b-35dd-4a98-8f29-d13c62b56206/router/0.log"